Review point on #447. The build-time check asserted macvlan, static and
host-local were executable but omitted dhcp, which the sibling
cni-dhcp-daemon DaemonSet runs and which its own comment says ships from
this installer ("the dhcp binary ships from the sibling
cni-plugins-installer DaemonSet ... whole tarball extracted").
Verified that dependency in kubernetes/infrastructure/multus/
cni-dhcp-daemon.yaml before changing anything, rather than taking the
review's word for it.
Consequence of the omission: since the whole /plugins directory is copied
regardless of what the loop checks, nothing is broken today — but the
"fail loudly if upstream restructures the archive" guarantee only covered
three of the four binaries this repo depends on. Upstream dropping dhcp
while keeping the others would have built green and surfaced later as a
broken DHCP daemon on whichever node rolled first.
Confirmed all four are executable in the already-published image, so this
tightens the assertion without changing what ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Supersedes the wget-instead-of-curl approach in #446, which removed the
`apk add` but left the real problems in place.
The installer fetched the containernetworking/plugins release tarball from
GitHub on the node and extracted it into /opt/cni/bin with NO integrity
verification of any kind. That directory is executed as root in the CNI
path on every node, so a tampered or MITM'd artifact would have obtained
privileged execution cluster-wide with nothing to detect it. It also put
github.com on the critical path of every node rebuild.
Now the binaries are baked into ci/images/cni-plugins, where the download
happens once and the sha256 IS verified, and the installer is a local file
copy with zero network calls (verified: no wget/curl/apk/https in the
rendered script).
The pinned digest was cross-checked against upstream's own published
cni-plugins-linux-amd64-v1.5.1.tgz.sha256 asset, not merely recorded from a
local download — a download can be tampered with, so comparing against
what upstream states is the point:
77baa2f669980a82255ffa2f2717de823992480271ee778aa51a9c60ae89ff9b
Verified the gate actually gates: with the correct digest the layer logic
downloads, verifies, extracts and asserts macvlan/static/host-local are
executable; with a deliberately wrong digest `sha256sum -c` FAILS and the
extract step is never reached.
Install script changes beyond dropping the fetch:
- Copies unconditionally instead of the old "already installed?" guard.
The guard existed to avoid re-downloading; a local copy is cheap, so
copying every time means a node always matches the version pinned in the
manifest rather than keeping whatever it first installed.
- Copies to a temp name in the same directory then rename(2)s over the
target. `cp` onto a plugin the kubelet is currently executing can fail
with ETXTBSY; rename never does and is atomic, so a concurrent CNI
invocation sees either the old or the new binary, never a partial one.
- LICENSE and README.md are no longer installed into /opt/cni/bin; the old
tar extraction put them there.
The chown 0:0 is kept but re-framed as a REPAIR. Its original cause is
gone — extraction applied the archive's `./` uid/gid (1001/127, the
github-actions runner) to the destination, and `cp` into an existing
directory does not touch its ownership. It still runs because nodes
provisioned by the old installer may already carry the wrong ownership,
which breaks Cilium's apply-sysctl-overwrites init container.
Verified end-to-end with the rendered script, including that repair path:
pre-seeded owner 1001:127 -> 0:0 after run
fresh install -> host-local/macvlan/static present, macvlan executes
and reports "CNI macvlan plugin v1.5.1", exit 0
re-run -> idempotent, exit 0, 18 plugins, no .tmp files left
Note the image carries ~80MB of extracted plugins (18 binaries), matching
what the old installer put on each node. All are kept rather than trimming
to the few in use, to avoid removing plugins already present on nodes as a
side effect of this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review point on #444, which merged before this could be folded in.
The comment claimed the listed shapes yield "null" at rc=0. That describes
BARE `.network_key.key`, not the expression actually written, and it
contradicted its own closing line two lines below. Measured against jq
1.7.1 in the image, `(.network_key?.key? // "")` returns an EMPTY string at
rc=0 for all of:
{} {"devices":[]} {"network_key":{}} {"network_key":null}
{"network_key":"abc"} {"network_key":123}
[] "a string"
and never the literal text null, because `// ""` maps a null result to
empty. Bare `.network_key.key` is what exits 5 on the last four shapes.
Checking that also turned up something the review did not mention: because
KEY can never be "null" with the current expression, the `[ "$KEY" =
"null" ]` arm is dead code today. Kept deliberately as belt-and-braces for
a future edit that drops the `// ""` fallback, and now labelled as such
rather than left looking like the arm that fires.
Comment-only: `git diff` shows zero changed lines outside comments, and the
rendered script is unchanged in behaviour.
NOTE: this text lives in the Deployment's init container command, so it is
part of the pod template — merging rolls the zigbee2mqtt pod (~20s of
Zigbee downtime, replicas: 1 + Recreate). Incidental benefit: the restart
resets Z2M's in-memory configure attempt counter, giving the Philips
devices that are interviewed but not yet bound (both RDM001 Treppenhaus
switches, the RWL022, and Sarah) three fresh configure attempts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round-3 review on #444. Both suggestions were optional; both were cheap to
close properly.
`.network_key.key?` only shielded the leaf selector. If the backup's TOP
level were not an object, `.network_key` itself raises before `?` applies.
Verified in the image with jq 1.7.1:
[] (.network_key.key? // "") -> rc=5 Cannot index array ...
"a string" (.network_key.key? // "") -> rc=5 Cannot index string ...
[] (.network_key?.key? // "") -> rc=0 empty
"a string" (.network_key?.key? // "") -> rc=0 empty
So `?` now sits on both selectors. Not a realistic shape for an Open
Coordinator Backup export and it failed closed either way, but it was the
last path that produced jq's raw error rather than a tailored FATAL line.
Full matrix against the rendered script, on the alpine-jq:2026-07-30 tag:
real backup -> exit 0, hash 27c5c77a945bcecc (matches live)
non-hex key -> FATAL: contains non-hex characters
4-char key -> FATAL: length 4, expected 32 hex chars
no network_key -> FATAL: .network_key.key absent
network_key non-object -> FATAL: .network_key.key absent
network_key number -> FATAL: .network_key.key absent
top-level array -> FATAL: .network_key.key absent
top-level string -> FATAL: .network_key.key absent
device link_key in doc -> correctly ignored; network_key used
On the registry suggestion: re-verified the tag pulls immediately before
merge — jq 1.7.1, alpine 3.21.7. It was published by pipeline 3060, the
push build for this branch. No build ran for the later commits because they
touch only kubernetes/apps/**, which correctly does not match the build
step's when.path; the image content is unchanged by them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round-2 review on #444.
The 🟡 identified a real gap, though not via the input shape it described.
Verified against jq 1.7.1 in the actual image:
{"devices":[]} -> "null", rc=0 (absent path is LENIENT)
{"network_key":{}} -> "null", rc=0
{"network_key":null} -> "null", rc=0
{"network_key":"abc"} -> rc=5, Cannot index string with string "key"
{"network_key":123} -> rc=5, Cannot index number with string "key"
So a missing network_key was already handled correctly and did produce the
intended FATAL line. What was NOT handled is a network_key that exists but
is not an object: plain `.key` exits 5 there, and under `set -e` that
aborts before the checks run, surfacing jq's raw error instead. Fixed with
`.network_key.key?`, which yields empty for those shapes and routes them
into the existing absent-key branch.
Now every malformed shape reports the same readable message:
real backup -> exit 0, hash 27c5c77a945bcecc (matches live)
non-hex key -> FATAL: contains non-hex characters
4-char key -> FATAL: length 4, expected 32 hex chars
no network_key -> FATAL: .network_key.key absent
network_key non-object -> FATAL: .network_key.key absent
network_key number -> FATAL: .network_key.key absent
device link_key in doc -> correctly ignored; network_key used
Verification ran against the alpine-jq:2026-07-30 tag itself.
On the 🔴: the premise does not hold — that tag already exists and is
pullable, verified twice by pulling it. The build step fires on branch
pushes as well as master, so pipeline 3060 published it when this branch
was pushed; the reviewer could not query the registry to see that. The
underlying caution is still worth recording, so the manifest now documents
that TAG is derived from `date` AT RUN TIME, that a date must never be
written here before a build has produced it, and that a later master build
publishes its own tag without deleting this one — so this keeps resolving,
though the image then came from a branch build rather than master. Also
recorded why it matters: Flux reconciles on a 10m poll with no ordering
against kaniko, and replicas: 1 + Recreate means an unpullable init image
would leave the sole coordinator in ImagePullBackOff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review point on #444. The reviewer cited a precedent I had missed:
ci-ansible:2026-04-29 in kubernetes/infrastructure/netbird-backup/
cronjob.yaml. Verified — for self-built images referenced by tag rather
than digest, this repo already uses DATED tags, with :latest used nowhere.
The argument holds independently of convention: with :latest, rebuilding
this base for some unrelated consumer silently changes what the seed-data
init container runs on its next pod restart, with no diff here to review.
That container renders the Zigbee network key, so "what exactly does it
run" should be visible in the manifest. A dated tag makes a change to it a
deliberate, reviewable act.
This does give up automatic pickup of rebuilds, which is the tradeoff the
previous commit chose the other way round. Given the base is thin and
rarely changes, and the consumer is security-sensitive, reviewability wins
— and it stays consistent with the owner directive that our own images do
not need digest pinning, since a dated tag is not a digest.
Verified gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30 pulls and
carries jq 1.7.1, so the reference resolves before merge rather than after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner directives, both applied here:
- "z2m-seed is a shit name for the image — name it for what it is, not for
how it is used." Renamed to alpine-jq. Nothing in it is
zigbee-specific; it is alpine plus jq, useful to anything that needs to
read JSON from a shell script. The Containerfile now describes it that
way, with the zigbee2mqtt case recorded as its first consumer rather
than its identity.
- "You do not need to pin the digest for our own images." Referenced by
tag. This also collapses what was going to be two PRs into one, since
the digest no longer has to exist before the Deployment can name it.
The seed-data init container now uses that image instead of
`apk add --no-cache jq` on every pod start.
Two consequences of dropping digest pinning, both good:
- The whole --image-name-with-digest-file / "PIN THIS:" apparatus is gone
from the pipeline. That machinery existed only to hand a digest to a
follow-up commit, and it is what broke every pipeline in the repo (#443)
via an unquoted ": " in a YAML scalar. Less surface, same outcome.
- :latest means Kubernetes defaults imagePullPolicy to Always, so a
rebuilt image is picked up without editing the manifest.
Kept the fail-loudly validation, since a wrong network_key costs the whole
Zigbee network: explicit checks for an absent key (jq prints "null" for a
missing path, so that is checked separately rather than being reported as
merely "non-hex"), non-hex characters, and a length other than 32.
Verified by running the manifest's RENDERED script, against the real
coordinator backup and synthetic fixtures. The alpine-jq tag does not
exist until this merges and builds, so the run used the existing image of
identical content (alpine 3.21.7 + jq 1.7.1, verified present):
real backup -> exit 0, network_key line byte-identical to the
live value (sha256 27c5c77a945bcecc)
non-hex key -> exit 1, "contains non-hex characters"
4-char key -> exit 1, "length 4, expected 32 hex chars"
absent network_key -> exit 1, ".network_key.key absent from ..."
device link_key also
present in the doc -> correctly ignored; the network_key is used
Validated with `woodpecker lint` (not just `yq`, which is what let #443
through), plus kustomize build and a server-side dry run. No stale
z2m-seed references remain.
NOTE: the now-orphaned z2m-seed package is still in the registry and
should be deleted; that needs package-delete rights this account does not
have.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
URGENT: this restores CI for the WHOLE repo. My commit 66dfd56b (merged
via #442) left an unquoted YAML scalar containing ": ", so Woodpecker
failed to compile the config:
yaml: construct errors: line 152: cannot unmarshal
'map[echo "PIN THIS:gitea.renner.uno/gitea_admin/z2m-seed@$DIGEST"]'
of type map[string]interface {} into a string value
Because a compile error rejects the entire pipeline set, every pipeline on
master has errored since that merge — not only build-ci-images. Confirmed
from pipeline history: 3048 (the push that introduced it) is the first
failure, and 3049-3053 follow, including an unrelated merge (#440) and the
alert-monitor cron.
Fixed by quoting the command as a single YAML scalar.
Why my pre-merge validation missed it: I checked with `yq '.' >/dev/null`,
which passed because the unquoted form IS valid YAML — it just parses as a
map instead of a string. Parsing successfully is not the same as matching
the schema. `woodpecker lint` catches it immediately and is now noted in a
comment at the line, along with the specific failure mode, so the trap is
not reinstated.
All 17 .woodpecker/*.yml files now lint clean apart from pre-existing
"Additional property timeout is not allowed" warnings in four deploy-*
files, which are untouched by this change and are warnings rather than
errors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review point on #442: kaniko writes one line per --destination to
--image-name-with-digest-file, so with both :$TAG and :latest the file
holds two entries carrying the same digest, and "PIN THIS:" would print
two near-identical lines for whoever does step 2 to choose between.
Rather than just comment on the ambiguity, remove it: reduce to the bare
digest, dedupe, and print one unambiguous copy-pasteable reference. The
multi-line behaviour is documented inline so a future edit does not
reintroduce it.
Verified the reduction against a two-line fixture matching kaniko's
format:
gitea.renner.uno/gitea_admin/z2m-seed:2026-07-30@sha256:abc123
gitea.renner.uno/gitea_admin/z2m-seed:latest@sha256:abc123
-> PIN THIS: gitea.renner.uno/gitea_admin/z2m-seed@sha256:abc123
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses the round-3 review (commit 39616401), which I failed to reply to
before pushing the next change.
Add an unconditional heartbeat logging the outstanding PENDING/FAILED
count. The main loop is purely event-driven, so if the
"Received payload: clusterID=" format ever changes (herdsman upgrade, log
format change) it would match nothing forever while still logging
"following <file>" and looking healthy. The per-frame recount cannot
detect that either, because it only runs when a frame is matched. The
heartbeat runs off a timer instead, so a count that never falls while
devices are demonstrably in use is a visible signal that the matcher
broke.
Document, in the ConfigMap itself rather than only in the PR body, that
the two log_* keys are seed-only and do not reach a running Z2M whose PVC
is already seeded. Both were therefore also applied to the live
configuration.yaml via bridge/request/options; verified they persist there
(lines 29-31 of the on-PVC file) so they survive a restart. Also noted
that log_symlink_current only takes effect on a Z2M restart, since the
symlink is created when the log directory is opened — until then the
sidecar waits and says so.
Not changed: resources.limits.cpu is still unset on the sidecar. The
review flagged it as consistent with the sibling containers in this file
and asked only whether it was intentional — it is. A CPU limit on a
mostly-idle process whose value is reacting inside a device's short wake
window buys nothing and risks throttling at exactly the wrong moment; the
memory limit already bounds the real risk.
Verified in eclipse-mosquitto:2.0.22 as uid 1883 with HEARTBEAT=5: the
heartbeat interleaves with frame handling and the simulated Z2M restart
without disturbing either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses the blocking review point on #441, which was correct.
`tail -F` follows a PATHNAME: it re-opens that name when the inode behind
it changes, but it can never notice that Z2M has started writing to a
different directory. Z2M creates log/<start-timestamp>/log.log fresh on
every start, so tailing the newest directory breaks as soon as Z2M
restarts — the old file merely goes quiet, so the pipe never ends, the
outer loop never re-resolves, "following <file>" still shows the stale
path, and the sidecar looks healthy while never seeing another frame.
Worse than the review assumed: this is not limited to the planned
"interview a batch, then restart Z2M once" step. The Z2M *container* has
restarted 38 times on its own from DNS flaps, each time into a new log
directory, while the pod (and so the sidecar) kept running.
Fixed at the source rather than with a polling watchdog: set
advanced.log_symlink_current: true so Z2M maintains log/current -> the
active directory, and follow that stable path. Verified that BusyBox
`tail -F` in eclipse-mosquitto:2.0.22 does follow a symlink repointed to a
new target mid-follow.
No fallback to the newest directory, deliberately: falling back could
silently tail a frozen file, which is the failure being fixed. A missing
symlink is instead reported loudly and repeatedly while waiting.
Also from this round:
- Short-circuit while nothing needs interviewing. The namespaced debug log
covers all of zh:controller, so every device on the network produces
frames; without this the script forked awk over database.db for each of
~30 devices' traffic forever. Recounted at most once a minute, so a
device regressing to PENDING is still picked up.
- Bound mosquitto_pub with `timeout`, so a hanging TCP connect cannot
stall the single-threaded read loop.
- Wire MQTT_HOST and MQTT_USER explicitly from the Deployment, like
BASE_TOPIC, instead of relying on the script's defaults.
- imagePullPolicy: IfNotPresent, matching the sibling container.
Two bugs found by testing this, not by inspection:
- `mosquitto_pub -W` does not exist (it is a mosquitto_sub flag). Adding
it made every publish fail on a usage error. Removed; `timeout` alone
bounds the call. Same class as the earlier `grep --line-buffered`
mistake, and again only visible by running it.
- `grep -c` prints "0" AND exits 1 when nothing matches, so
`grep -c ... || echo 0` emitted two lines ("0\n0") and every `-eq` test
died with "bad number". This surfaced only once no devices needed an
interview — the steady end state this feature exists to reach.
Verified in eclipse-mosquitto:2.0.22 as uid 1883, simulating Z2M's layout
and a restart (new directory + repointed symlink):
frame before restart -> interview requested
symlink repointed to a new directory
frame after restart -> interview requested <- the reviewed failure case
zero-pending fixture -> "no devices need an interview; idling", no errors
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 1 of 2 toward removing `apk add` from the zigbee2mqtt seed-data init
container, which currently installs jq on every pod start. Owner
directive: baking tools into a layered image is fine, adding them at
runtime is not.
This commit only adds the image and its build step; nothing consumes it
yet, so it is inert on merge. The Deployment cannot be switched in the
same commit because runtime images in this repo are pinned by DIGEST (as
the secret-rotator CronJobs are), and the digest does not exist until
this has built and pushed. The build therefore writes it out via kaniko's
--image-name-with-digest-file and echoes it as "PIN THIS:", so the
follow-up commit can copy the reference straight from the build log
without a separate registry query.
Why a real JSON parser rather than busybox sed/awk: "key" occurs 12 times
in the actual backup — once per device with an APS link_key — and the
FIRST occurrence is a device key, because "devices" sorts before
"network_key". An extraction that is not correctly anchored yields a
device link key, herdsman then forms a NEW network, and every device
drops off. This was reproduced while prototyping the awk approach, which
is what makes jq worth an image here rather than clever one-liners in the
manifest.
Follows the existing conventions: Containerfile alongside the other
ci/images entries, base pinned by digest to the same alpine:3.21 the
ci-k8s image uses, kaniko build gated on its own path so unrelated pushes
do not rebuild it, and date + latest tags pushed as the other images do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner directive: `apk add` at container start is not acceptable.
Replace alpine:3.21 + `apk add mosquitto-clients` with the broker image
eclipse-mosquitto:2.0.22, used purely for its bundled mosquitto_pub. No
packages are installed at runtime, so the sidecar no longer depends on
network or package-index availability at start.
Every tool the script needs was verified present in that exact image
rather than assumed: sh, tail (including -F, which BusyBox does support),
awk with match()/RSTART/RLENGTH, cat, date, mkdir, mosquitto_pub. The awk
extraction was re-run there and resolves a PENDING device, skips a
SUCCESSFUL one, and skips an unknown address.
Pinned WITHOUT a Flux $imagepolicy marker on purpose: this sidecar is
temporary and must not drag the broker's image-automation policy with it.
Now genuinely non-root (1883/1883, the image's own mosquitto user). The
previous commit justified running as root because `apk add` needed it;
with that gone the only remaining requirement is reading /app/data, and
those paths are world-readable (dirs 0755, files 0644 — verified against
the live PVC), so runAsNonRoot is viable. Cooldown stamps moved to an
explicit emptyDir at /state rather than the image's /tmp, so they stay
writable as non-root without relying on the container filesystem.
Re-verified end-to-end in this image as uid 1883 with no package install:
running as: uid=1883(mosquitto) gid=1883(mosquitto)
interview-watch: 1 device(s) currently need an interview
interview-watch: following /app/data/log/.../log.log
interview-watch: frame from 0xdeadbeefdeadbeef (nwk=65001 cluster=6) ...
interview-watch: interview requested for 0xdeadbeefdeadbeef
with the negative paths also holding: a frame from an already-SUCCESSFUL
device is ignored, and a malformed address neither fires nor crashes.
NOTE: the pre-existing seed-data init container still runs
`apk add --no-cache jq` (line 49). Untouched here as it is out of scope
for this PR, but it violates the same directive and is raised separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second review round on #441. The first point was a real blocking defect
that would have shipped silently.
BusyBox grep does not support --line-buffered. Confirmed by running it:
$ grep --line-buffered -oE '...' <<< 'Received payload: ...'
grep: unrecognized option: line-buffered
GREP_EXIT=2
The sidecar runs alpine:3.21 and `apk add mosquitto-clients` does not
pull GNU grep, so the `tail -F | grep --line-buffered | while read` pipe
would have produced zero lines forever while the container still logged
"following <file>" and appeared healthy — a silent no-op of the entire
feature, undetectable from the diff.
Fixed by removing the grep stage rather than installing GNU grep: match
in the read loop with a `case` glob and extract the fields with parameter
expansion. `tail` writes per line and `read` consumes per line, so there
is no buffering question and no extra package. Added a numeric guard on
the extracted address so a future log-format change cannot feed garbage
downstream.
Also from this round:
- Write the cooldown stamp only on a SUCCESSFUL publish. Stamping before
the result was known burned a device's wake window whenever
mosquitto_pub failed, then refused to retry for the full 300s despite
no interview having been requested.
- Derive the request topic from BASE_TOPIC (env, defaulting to
zigbee2mqtt) instead of hardcoding it, so it cannot silently drift from
mqtt.base_topic. Noted that the MQTT ACL is scoped to zigbee2mqtt/# and
would need changing too.
- Harden the sidecar with allowPrivilegeEscalation: false and
capabilities drop ALL. Deliberately no runAsNonRoot: apk add needs
root, and Z2M writes database.db and the log dir as root, so a non-root
uid could not read either input.
- Runbook callout said "Step 2" where it meant item 2 within Step 5.
Verified end-to-end in an alpine:3.21 pod against this script (the
earlier "verified live" evidence came from the session prototype, which
resolved friendly names and is not this code — see PR discussion):
counts only PENDING devices, fires on a frame from a PENDING device with
mosquitto_pub returning success, ignores a frame from an
already-SUCCESSFUL device, and survives a malformed address without
firing or crashing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review points from review-bot on #441.
Stay resident instead of exiting on log-stream end. Previously the
sidecar exit 1'd so the kubelet would restart it and re-resolve the log
file, but that re-ran `apk add mosquitto-clients` over the network every
time and, worse, incurred kubelet crash-loop backoff — which climbs to
5 minutes unless the container stays up >=10 min. Z2M restarts and log
rotations land closer together than that (notably during the "interview
a batch, then restart Z2M once" workflow), so backoff would have kicked
in exactly when a device's short wake window needs a prompt reaction.
The log file is now re-resolved in an outer loop, which also drops the
post-startup network dependency. Chosen over baking a custom image
because it removes the restart entirely rather than making restarts
cheaper.
Match ieeeAddr/nwkAddr independently rather than with one combined sed
pattern. JSON does not guarantee key order, so the old pattern's
assumption that "ieeeAddr" precedes "nwkAddr" held only by virtue of
herdsman's current serialization. Had that changed, the regex would
capture nothing and a device would silently never be picked up — no
error, no log. Now extracted per-field with awk match(), with the
rationale recorded in a comment. Verified against the live database.db
under busybox awk (the sidecar runs alpine): resolves both still-pending
devices, and correctly returns nothing for one that has since been
interviewed.
Flag Step 5 of the migration runbook as automated, so operators do not
hand-chase devices the sidecar already handles. Includes the two
caveats that cost real debugging time: interviewing does not bind
manuSpecificPhilips (needs one Z2M restart), and a device that never
transmits is offline rather than un-interviewed — check lastSeen before
hunting a Z2M-side fault, because a flat battery presents identically.
Not changed: mosquitto_pub takes a password only via argv (-P, or an -L
URL) — it has no --pw-file or env option, verified against the client in
the cluster — so the 🟢 nit cannot be resolved with this client. Not a
cross-container exposure, as the pod does not set shareProcessNamespace;
noted in a comment at the call site.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ZHA->Z2M cutover adopted the existing Zigbee network, so devices
stayed joined but database.db started empty — every device needs an
interview before Z2M knows its modelId and can match a converter.
Mains routers interviewed immediately. 16 battery end devices did not,
and they do not self-heal: zigbee-herdsman only retries an interview on
a ZDO device announce, which sleepy devices essentially never emit. They
sat PENDING for six days across 38 process restarts, exposing only a
generic "Automatically generated definition" (battery/linkquality)
instead of action/contact/moisture entities.
A device that transmits is awake, which is exactly when an interview can
succeed. The sidecar watches Z2M's log for the zh:controller frame-receipt
line and fires a forced interview for any device still PENDING/FAILED.
Verified in practice: interviews completed in 11-34s, including
Xiaomi/Aqara sensors, and two fired with no human interaction at all
(a periodic OTA check and an illuminance report).
Implementation notes:
- Sidecar, not a separate Deployment: it reads Z2M's log file and
database.db, and the PVC is RWO/single-writer. Mounted read-only.
- Signal comes from the log file, not MQTT: an un-interviewed Xiaomi
device publishes nothing at all (its manufacturer-specific attributes
cannot be mapped by the auto-generated definition), and
bridge/logging carries only info-level messages.
- Needs advanced.log_namespaced_levels {zh:controller: debug}. Namespaced
rather than global debug on purpose: global adds the unpi:parser
byte-level firehose (~90 lines/min measured) vs ~6 lines/min here.
- database.db is newline-delimited JSON despite the extension, so the
nwk->IEEE lookup needs no jq and is re-read per frame, which keeps it
correct across network-address changes and drops devices as they pass.
Interviewing is only half the job: Philips devices also need configure to
bind manuSpecificPhilips (0xFC00), and that cluster is registered only on
Z2M's start event, so a device interviewed mid-run stays unbound until the
next restart. One restart after a batch of interviews completes the binds.
Both the sidecar and the namespaced log level are temporary — remove them
once every device is interviewed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Root cause of #393, confirmed. Three things combine:
1. The public renner.uno zone has a *.renner.uno CNAME to proxy.renner.uno
that matches at EVERY depth, so renner.uno never returns NXDOMAIN:
definitely-not-real-xyz123.iot.renner.uno -> proxy.renner.uno -> 45.157.177.26
Any name that fails locally resolves publicly to the WAN proxy instead of
failing honestly.
2. TTL asymmetry. The correct answer is a dnsmasq DHCP-lease record with
TTL 5 (verified: `dig @10.2.50.1 shellyplusi4-....iot.renner.uno` returns
`aa` + TTL 5). The wrong wildcard answer has TTL 300 — 60x more durable.
So one 5-second failure pins a wrong answer for 5 minutes. That is exactly
the "correct at 14:10, wildcard-only at 15:45" report in the issue.
3. The UDM's dnsmasq refuses connections in bursts — 330 "connection refused"
errors in the last 3000 k8s-gateway log lines, on BOTH 10.2.40.1 and
fd00::1.
Note the issue's premise was wrong: there is no "pinning job". The device
records are ephemeral DHCP-lease records in dnsmasq, not IaC. external-dns-unifi
is also NOT the culprit despite running --policy=sync: it writes the UDM's
static-DNS list, not the lease table, and --registry=txt confines deletion to
records carrying its own TXT.
Fix: raise the cache floor and serve through upstream outages.
success ... 60 clamps the 5s TTL up to a 60s floor
serve_stale 1h immediate keeps answering while the upstream refuses
This also fixes the zigbee2mqtt crash-loop, which has the same cause — z2m
exits on `getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno` with no retry, and the
k8s-gateway log shows that exact name failing against both UDM addresses.
Validated before committing, because CoreDNS is cluster-critical and the
`template` plugin previously broke cluster DNS here (7a09bad5):
- The repo's full Corefile runs under the real coredns:v1.13.2 image (parses;
only the expected out-of-cluster kubernetes-plugin error).
- The repo's k8s-gateway values render via `helm template` (rc=0) and the
resulting Corefile starts under the real k8s_gateway:1.7.0 image
(CoreDNS 1.14.2).
- The cache parameters MUST stay on one line: a multi-line value breaks the
chart's ConfigMap render with "could not find expected ':'". Confirmed by
hitting it.
- The forward block is deliberately untouched — its options require newlines
and cannot be expressed through the chart's single-line parameters.
- The `template` plugin is NOT reintroduced.
Known not fixed: AAAA for lease-backed names is deterministically shadowed
(dnsmasq has an A from the lease but no AAAA, so AAAA forwards upstream and
hits the wildcard every time). That needs a NODATA response for those names
and is a separate, tightly-scoped change — filing separately.
Refs #393
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There are zero ceph_* series in Prometheus, and there have been for all 79
days of the cluster's life:
count of ceph_ metrics in Prometheus: 0
rook-ceph active scrape targets: 0
Not a Rook misconfiguration — `monitoring.enabled: true` is set and both
ServiceMonitors exist. They are simply never selected:
Prometheus serviceMonitorSelector : matchLabels {release: kube-prometheus-stack}
rook-ceph-mgr / rook-ceph-exporter: labels {team: rook}
The Rook operator creates those ServiceMonitors at runtime with its own
hardcoded label, so no static manifest or Kustomize patch in this repo can
relabel them.
Cluster-wide this was dropping 12 of 30 ServiceMonitors:
rook-ceph/rook-ceph-mgr, rook-ceph/rook-ceph-exporter (team: rook)
authentik/ak-outpost-* x10 (goauthentik.io/...)
Consequences: the entire storage layer had no telemetry and no history — so
Ceph trend analysis is impossible for anything before this lands, and any Ceph
PrometheusRule added today would have silently had nothing to fire on.
This is the ServiceMonitor twin of the trap already documented for rules in
monitoring-rules/prometheusrule-velero.yaml ("without this label the rule is
created but never loaded into Prometheus").
ruleSelector is deliberately NOT loosened — this repo's PrometheusRules all
carry the release label on purpose, and keeping that selector strict stops a
stray third-party rule from firing here.
Blast radius: 12 new scrape targets. The 10 authentik outposts come along with
the 2 rook ones; they are legitimate targets, but they are a side effect of
this change rather than its goal.
Found while investigating the Ceph BLUESTORE_SLOW_OP_ALERT warning, which
itself turned out to be a benign self-limiting event.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two v6 addresses in hairpin_endpoints were attributed to the opposite
proxy members. DNS is the operational truth, verified independently:
getent ahosts proxy-1.renner.uno -> 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29
(v6-only member, single AAAA)
getent ahosts proxy-2.renner.uno -> 2a03:4000:4b:c1e:4474:43ff:fedf:8f85
45.157.177.26 (dual-stack member)
Purely a labelling fix with zero functional impact: both entries DNAT to the
same target (fd99::86), so the hairpin behaved identically before and after.
Slots are untouched — the file's "never reuse a freed slot" rule applies to
the keys, not the values.
Also trims the now-stale NOTE in probes-blackbox.yaml that existed only to warn
readers away from the wrong labels; the two files now agree.
Surfaced while triaging #392/#420, both of which turned out to be already
fixed. `tofu validate` passes. (Note: `tofu fmt` reports pre-existing
alignment drift further down this file, at the jsonencode block around line
108 — present on master before this change and deliberately left alone.)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The velero e2e check manufactured phantom PartiallyFailed Backups that then
polluted backup-health triage. Sequence, from Loki for e2e-1785031248:
02:01:05 Backup + PVC created (DataUpload -> phase Prepared)
... data path never starts (node-agent slot held elsewhere)
~02:06 the 60x5s = 300s wait expires -> check FAIL, but does NOT abort
-> script deletes ns velero-e2e and creates the Restore
02:07:03 restore error "no DataUpload result cm found" -> PartiallyFailed
02:10:08 script deletes Backup+Restore -> "Cancel du under phase Prepared
because it is being deleted" -> Backup lands PartiallyFailed
Both 2026-07-26 occurrences (02:00:55 and 12:04:05) fall inside a running
restore-test sweep. node-agent-config.yaml sets no loadConcurrency, so Velero's
default of one data path per node applies and the sweep's data-mover pods hold
the slot — a 1 GiB e2e DataUpload then sits in Prepared well past 5 minutes.
The backup was never actually faulty; the test cancelled it.
Three changes:
- Wait 120x15s (30 min) instead of 60x5s (5 min), so a queued DataUpload has
time to run once the slot frees.
- Guard the destructive half (namespace teardown + restore roundtrip) behind
whether the backup check actually passed. Tearing down while the upload is
in flight is what cancels it.
- Do not delete the Backup on the failure path either — that is the same race
relocated to cleanup. The Backup carries ttl: 2h0m0s, so Velero's GC reaps
it regardless and leaving it costs nothing.
Cleanup of the restore, namespace and released PVs stays unconditional, so
nothing leaks. Verified with `bash -n`.
Found while triaging #427; the restore-test failure there was unrelated and
is already fixed.
Refs #427
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The alert triage agents could reach Prometheus and Alertmanager through the
apiserver service proxy, but not Loki:
kubectl auth can-i get "services/loki:3100" --subresource=proxy -n monitoring \
--as=system:serviceaccount:monitoring:ci-diagnostics -> no
(same check for kube-prometheus-stack-prometheus:9090 -> yes)
That gap has a sharp edge. A KubeJobFailed-class alert can only be cleared by
DELETING the failed Job — the rule's own annotation instructs this — and the
deletion destroys the pod and its logs. So the agents would inspect a failed
Job, find no live pod, declare the root cause "unconfirmable", and then delete
the last copy of the evidence.
The evidence was never actually gone: Alloy ships all pod logs to Loki with
retention_period: 2160h (90d). Issue #380 declared velero/restore-test-29740440
"probably unconfirmable" and its full failure output was in Loki the whole time
(FAIL: 3 — apps restore phase + two PVC parity checks).
Adds `loki` / `loki:3100` to the existing service-proxy Role (read-only GET,
same bare + :port-qualified pattern the other entries use, verified against the
live Service which exposes http-metrics:3100), and teaches both agent prompts
the query recipe so "the pod is gone" stops being accepted as a dead end.
No new privilege class: the agents already hold pods/log cluster-wide. This
mainly extends their reach to *deleted* pods.
Refs #427, #380
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `>=1.0.0 <2.0.0` ceiling had reached its practical maximum: chart 1.3.0
(app v1.3.0) is the last 1.x ever published, so the bound was holding the
cluster on a dead line while upstream had moved on to app v1.4.19.
Verified before bumping:
- The only breaking change at the 1.x->2.0.0 boundary is the relocation of
`deployment.image.*` to top-level `image.*`. This HelmRelease sets
`values: {}`, so neither key is in play and the bump is a drop-in.
- `helm template stakater/reloader --version 2.2.14` renders cleanly with
default values (rc=0, image v1.4.19).
- ClusterRole rules are byte-identical between 1.3.0 and 2.2.14 — the
cluster-wide privileges of this restart-authority controller do not change.
2.x additionally ships a namespace-scoped Role (configmaps
list/get/watch/create/update) used for leader election. Additive only.
- `kubectl kustomize kubernetes/infrastructure` renders 199 resources with the
HelmRelease resolving to the new constraint.
Floored at 2.2.0 rather than 2.0.0 to skip the early 2.0/2.1 line outright.
Resolves to 2.2.14 today.
Blast radius on upgrade: reloader itself rolls; the 7 annotated workloads
(apps/external-dns, apps/woodpecker-agent, apps/woodpecker-server,
home-assistant/home-assistant, home-assistant/shelly-manager,
monitoring/pve-exporter, paperless/paperless) are not restarted by a
controller upgrade alone.
Refs #426
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to #416 (ceph-block-ephemeral) for restore-test.sh's own PV leak
(incident 2026-05-17, ~2.7 TB of orphan CephFS subvolumes).
The intent was to retire purge_released_pvs entirely. That is not safely
possible, and the reason is worth recording. Three categories of throwaway
PVC exist per iteration; only one is ours to pick a StorageClass for:
1. source snapshot-probe PVCs (rt-src-*) — this script creates them.
FIXED HERE: the CephFS branch of sc_for_snapshot_probe returned
`ceph-filesystem` (Retain), so every probe leaked a Released PV. It
now returns `ceph-filesystem-ephemeral` (Delete). The two classes are
identical apart from reclaimPolicy — same provisioner, parameters and
mountOptions — so the snapshot restores into it unchanged.
The ceph-block branch needed no change and must not get one: it
already uses ceph-block-snapshot-readonly, which is Delete, and whose
`noload` mountOption is load-bearing (skips ext4 journal replay on the
RO snapshot mount). ceph-block-ephemeral lacks noload, so pointing the
probe at it would reintroduce the dirty-journal mount failure.
2. destination PVCs in *-restore-test namespaces — Velero creates them
from the backed-up PVC spec, so the class is whatever production uses
(Retain). The only remap knob is the change-storage-class
RestoreItemAction ConfigMap, which Velero applies to EVERY restore in
the installation with no per-restore scoping. Using it would strip
Retain from a real disaster-recovery restore. Rejected.
3. Velero data-mover staging PVCs in velero/ (*-rtest-*) — derived from
the target PVC. The node-agent restorePVC config supports only
`ignoreDelayBinding`; unlike backupPVC there is no storageClass field
on Velero v1.18.
So 2 and 3 keep purge_released_pvs load-bearing; it stays, now documented.
Also fixes a real residual leak the ephemeral class does not address:
cleanup_service's purge is scoped to ${RUN_ID}, which is right during a run
but means a run that dies before cleanup (activeDeadlineSeconds, OOM, node
eviction, killed Job) strands its PVs forever — no later run's pattern ever
matches an older RUN_ID. Adds an unscoped sweep at run start. Safe because
concurrencyPolicy Forbid guarantees no other restore-test Job is active, and
it only ever touches Released PVs whose claimRef matches restore-test-only
naming patterns.
Verified against the live cluster: the sweep pattern matches zero PVs today,
and in particular does not match the three intentional Retain safety PVs
(crowdsec/crowdsec-db-pvc, openbao/data-openbao-0,
home-assistant/mosquitto-data). Categories 2 and 3 were confirmed to
provision on Retain classes by observing an in-flight run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- No TOFU: the CronJob writes a pinned known_hosts (ed25519 key captured
via ssh-keyscan 2026-07-26 and cross-checked against the workstation's
trusted entry) and uses StrictHostKeyChecking=yes. A reprovisioned VPS
makes the job fail loudly — desired.
- The dump now contains ONLY store.db: /opt/netbird/config is fully
regenerable from the role's templates + vault, and including it put
the plaintext TURN secret into every nightly tarball. Restore is
role-run (rebuilds config) + untar into /opt/netbird/management/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The restart could in principle deconfigure the interface and cut the
SSH connection Ansible rides on (dhcpcd default without persistent).
Verified live: persistent is in fact already set on both members (line
19), so the risk was theoretical here — but 'dhcpcd -n' (reload config
+ rebind, verified supported) achieves the same effect with no
deconfigure window at all, and the role now asserts persistent
explicitly instead of relying on it being there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses the automated review on #418.
Closes the review's open question about whether the -prune/-path/paren
grouping behaves the same in the probe pod's own busybox build. Rather
than leaving it as a "watch the first real run" note, it was tested
directly: an alpine:3.21 pod (BusyBox v1.37.0) with the same
securityContext and read-only mount as count_files_in_pvc's probe,
against the live audiobookshelf-metadata volume.
unfiltered: 2339
pruned streams + tmp: 775 == 2339 - 1531 - 33
Two robustness properties also confirmed in that image:
- pruning a path that does not exist on the volume is a no-op with
exit 0, so a stale entry degrades gracefully instead of zeroing the
count and faking a catastrophic-loss FAIL;
- each prune subtracts exactly its own subtree (tmp alone -> 2306).
Also renames the local `expr` to `find_expr` so it cannot be misread as
the expr(1) command by someone skimming for invocations. No behaviour
change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- The blocking catch: the openbao ClusterSecretStore's namespace
conditions did not include the new netbird-backup namespace, so the
ExternalSecret could never sync even after key provisioning. Added
(sorted, with consumer comment).
- rm -f /backup/*.part up front — aborted-run leftovers are outside the
retention glob.
- storageClassName: ceph-block explicit, matching the repo's other
single-PVC manifests.
Keeping the key on root with restrict,command= rather than a dedicated
VPS user: the forced command is the actual privilege boundary (the key
can only ever produce a dump), and a separate user would need read
access to root-owned live SQLite + config either via group surgery on
/opt/netbird or sudo rules — more moving parts for the same effective
capability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The review flagged the v6 target as 'proxy-2 per unifi_nat_hairpin.tf'.
The tf label keys are the swapped ones: DNS (dig) and the live hosts
(the v6-only member with the nameserver-less dhcpcd lease answers as
proxy-1.renner.uno = 2a03:…:28:f6e…; the dual-stack member answers as
proxy-2.renner.uno = 45.157.177.26 + 2a03:…:4b:c1e…) both agree with
the probe's original attribution, as does the netbird-proxy role's
'proxy-1 is v6-only' doc. Keep DNS attribution, document the tf
mislabel (tracked for a separate relabel — renaming those tf map keys
churns NAT rule resources, so it doesn't belong in this PR), and:
- add proxy-2's v6 address as a third target (previously its v6 path
was unprobed; verified live from a pod, returns 200)
- pin job="blackbox-http" on NetbirdControllerUnreachable for
consistency with the other rules
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- insertbefore: BOF so nohook stays global if an interface stanza ever
appears (none exist today, verified live on both members).
- dhcpcd restart is now an inline when-changed task instead of a
handler: nohook must take effect before a lease renewal can re-clobber
the pinned file, and meta: flush_handlers would also fire the
proxy-recreate handler before the compose file exists on a fresh host.
Unit name verified as plain dhcpcd.service on both members.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-07-26 restore-test run failed 1 of 40 checks:
audiobookshelf: PVC audiobookshelf-metadata file count parity (±10%)
-- source=2339 restored=803
This was a false positive. The backup was complete: backup
audiobookshelf-daily-20260725034518 is phase=Completed with no errors or
warnings, and all four of its DataUploads finished with bytesDone ==
totalBytes.
The parity check compares a backup taken up to ~24 h ago against the live
volume *now*, counting files with an unfiltered `find`. Between the backup
(07-25 03:45) and the source probe (07-26 02:12), one Audiobookshelf
playback session wrote 1531 HLS transcode files (~153 MiB) into
metadata/streams/. Excluding that directory: 2339 - 1531 = 808 persistent
files vs 803 restored, a 0.6% delta — comfortably inside the tolerance.
The next day's metadata DataUpload was +157 MB, independently confirming
the stream segments were the only thing that changed.
Fixes:
1. Add volatile_paths_for(), a per-"<namespace>/<pvc>" list of transient
subtrees pruned from BOTH the source and restored counts. Only
streams/ and tmp/ are excluded for audiobookshelf-metadata; cache/ is
left counted deliberately, since it is stable enough not to threaten
the tolerance and dropping it would cost real coverage. Every entry is
a blind spot, so the list is kept minimal. Volumes with no entry keep
the exact previous behaviour.
2. Move the CronJob from Sunday 02:00 to Sunday 08:00 UTC. The old slot
was picked as "well before Monday's daily backups", which missed that
the daily schedules also fire on Sunday — a ~5 h sweep starting at
02:00 sat on top of the whole 03:15-07:15 backup window and starved it
of node-agent DataUpload concurrency. audiobookshelf-daily took 2 h
18 m (07-19) and 2 h 33 m (07-26) against a ~1 min baseline, with the
books DataUpload not starting until 06:18. Both Sundays; no other day
shows it.
The pruning find expression was verified against the live
audiobookshelf-metadata volume (busybox find, same as the alpine probe
image): it returns 775, matching 2339 - 1531 - 33.
Refs #380.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- e2e-test.sh: note that purge_released_pvs '^default/e2e-pvc$' is now a
one-time safety net (e2e-pvc uses ceph-block-ephemeral, so it normally
no-ops) rather than dead weight.
- RECOVERY.md: explain why Delete reclaim is safe *here* specifically — this is
the throwaway restore-test, never a real DR target (a real cold-rebuild uses
the live cluster's Retain ceph-block storage).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the last big SPOF-assessment item: the management SQLite store
(peers, setup keys, groups, routes) had zero backup coverage — losing
the Hetzner VPS meant re-enrolling the entire mesh. The VPS cannot
reach any PBS (both PBS names resolve to the public proxy, ports
closed), so the cluster pulls instead of the VPS pushing:
- VPS (netbird-server role): a dump script takes a CONSISTENT snapshot
via sqlite3 .backup (online-backup API, safe against the live writer)
plus /opt/netbird/config and streams a tar.gz to stdout. A dedicated
ed25519 key is authorized for root restricted to exactly that script
(restrict,command=…) — the puller can run nothing else.
- Cluster (kubernetes/infrastructure/netbird-backup): nightly CronJob
(02:45) ssh-pulls the dump onto a 5Gi Ceph PVC, gzip-verifies before
renaming, keeps 14 dumps; ExternalSecret syncs the private key from
OpenBao (homelab/infrastructure/netbird-backup, one-time manual
provisioning like the rotator bootstraps); restricted-profile pod.
- Offsite: new Velero Schedule netbird-backup-daily (07:45, PVC-only,
snapshotMoveData, 30 d TTL) ships the PVC to Hetzner Object Storage.
- Alerts: NetbirdVpsBackupStale (critical, >26 h without success) and
NetbirdVpsBackupNeverRan (warning — catches the unprovisioned-secret
state so the backup cannot die silently).
Restore: untar over /opt/netbird/{management,config} after the role
rebuilds the stack.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the daily RBD-image leak (found while reclaiming 15 orphaned
images on 2026-07-26): throwaway block PVCs used the default ceph-block
StorageClass, which is reclaimPolicy: Retain. On PVC deletion the PV goes
Released (image retained), and the ad-hoc reclaim (patch reclaimPolicy=Delete ->
delete pv) races the ceph-csi provisioner — if it acts on the stale Retain
policy it removes the PV but orphans the RBD image. Same class of bug as #262,
which was already fixed for CephFS via ceph-filesystem-ephemeral but never for
RBD.
- Add ceph-block-ephemeral (reclaimPolicy: Delete), the RBD mirror of the
existing ceph-filesystem-ephemeral, in rook-ceph-cluster/storageclass.yaml.
- e2e-test.sh: point the throwaway default/e2e-pvc at it (this was the ~1
image/day source via the daily e2e crons).
- RECOVERY.md: point the CNPG restore-test drill's throwaway cluster at it and
simplify the teardown — with Delete reclaim, deleting the cluster removes the
PVC -> PV -> RBD image automatically, so the manual PV-reclaim dance is gone.
Residual (separate): the velero restore-test (Sundays) restores PVCs with the
source's storageClass (ceph-block, Retain) and can't be repointed the same way;
it still relies on its purge_released_pvs helper.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the vantage-point gap that let the 2026-07-24 controller outage
run ~20 h unnoticed: every existing probe of the public names resolves
in-cluster (CoreDNS/k8s_gateway) or gets hairpinned, so nothing measured
the actual WAN path.
- blackbox module http_2xx_wan_edge: probes the edge by proxy IP
literal with SNI/Host pinned to home.renner.uno. The Server/Ingress
VLANs are deliberately absent from the UniFi hairpin, so IP-literal
probes from the in-cluster exporter genuinely leave via the WAN and
come back through the netcup edge (verified live from a pod: v4
returns 200; v6 currently fails against proxy-1 — real breakage,
fixed separately by the resolv.conf PR).
- New probes: WAN edge per member/family (proxy-2 v4, proxy-1 v6),
https://netbird.renner.uno (controller), relay :9000/health on both
netcup members, tcp :33080 relay listeners.
- New PrometheusRule netbird-availability: controller unreachable
(critical 10m), whole edge down from WAN (critical 5m), single edge
member down (warning 15m), relay member down (warning 15m).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dhcpcd's resolv.conf hook overwrites the systemd-resolved stub file that
/etc/resolv.conf symlinks to. On v6-only proxy-1 the lease carries no
resolver options, leaving the file with zero nameserver lines; host
tools keep working via nss-resolve, but the host-network proxy container
copies that file at create time and Go's resolver falls back to
localhost:53. Result: proxy-1's reverse-proxy has been unable to reach
management since the host-network switch — no cert mapping, TLS
'unknown domain "home.renner.uno"' for v6 edge clients that resolve to
its address. Bridge mode had masked the broken file via Docker's
embedded DNS; proxy-2 only survived because its v4 lease happened to
write usable nameservers into the same clobbered file.
Fix on both members: nohook resolv.conf for dhcpcd (+ restart handler so
it takes effect immediately) and a static /etc/resolv.conf pointing at
the resolved stub (127.0.0.53, healthy netcup upstreams on both hosts),
then force-recreate the proxy so it picks up the fixed file.
Found by the WAN-path edge probe work: the v4 probe returned 200 while
the v6 probe failed TLS against proxy-1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The seed->apps ownership handoffs are complete, so the prune: disabled comments
on the folded CNPG DB objects no longer read 'ahead of the ... handoff' (future
tense). Normalize the 17 comments across authentik/immich/paperless/woodpecker/
influxdb to standing-guard/past tense. Comment-only; no manifest changes.
Closes the tense-normalization item flagged on the immich fold review (#405).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review follow-up: note in the compose template why the relay is not
covered by the proxy watchdog (fallback component, client-side relay
failover, restart: unless-stopped) and record the :9000/health endpoint
(verified on 0.73.2) for future active monitoring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second/third NetBird relay (netbirdio/relay, host network, ws :33080,
embedded STUN :3478/udp) on proxy-1/proxy-2, sharing the existing relay
auth secret, and registered in management.json Relay.Addresses and
Stuns. Clients pick a relay at random and fail over automatically, so
relayed connections and — via the added STUN — new P2P candidate
discovery now survive a controller-VPS (Hetzner) outage. Closes
mitigation option 2 from the 2026-07-25 SPOF assessment.
Plaintext rel:// is deliberate: :443/:80 on these hosts belong to the
reverse-proxy and redirect (no HTTP-01/ALPN slot), and a zone-wide
DNS-01 token on an internet-facing host is a worse trade. Peer payload
stays WireGuard-encrypted end to end; exposure is relay framing
metadata and credential sniff-and-reuse bounded by the 24h TTL.
proxy-1 is v6-only — v4-only clients that draw it fail over to the
dual-stack members.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The last app in the seed->apps migration. Folds authentik's ak-pg CNPG backend
into the existing dedicated authentik KS and, since authentik was the final
holdout, fully retires kubernetes/seed/.
- git mv seed/authentik/{cluster,objectstore,scheduled-backup,barman-creds-
externalsecret,hetzner-creds-externalsecret}.yaml -> apps/authentik/, add to
apps/authentik/kustomization.yaml (authentik has no data PVCs — all state is
in ak-pg).
- authentik KS: drop 'dependsOn: seed-authentik', add
'dependsOn: infrastructure-cnpg-barman', add the CNPG Cluster healthCheckExprs.
- Delete clusters/homelab/seed-authentik.yaml and the now-empty seed/authentik/.
Seed-layer retirement cleanup (kubernetes/seed/ is now empty):
- git mv seed/RECOVERY.md -> kubernetes/RECOVERY.md (sibling to the MIGRATION-*
docs); fix its stale 'flux reconcile kustomization seed-authentik' example to
'authentik'.
- e2e-test.sh: drop the last seed-* readiness check (seed-authentik).
- bootstrap.sh: rewrite the obsolete 'per-app seed-<app> KSes' gating comment to
reflect that app KSes now own their CNPG DBs directly, and de-'seed' the
wait-description labels (the one-shot rsync/pg_restore seed Jobs are long gone).
- opentofu/object_storage.tf: fix the CNPG ObjectStore path comment
(kubernetes/seed/* -> kubernetes/apps/*).
Safe ownership transfer (nothing pruned): all 5 seed/authentik objects carry
prune: disabled (from stage 1 #407), so deleting seed-authentik skips them in GC
and the authentik KS adopts them in place. No CNPG restart, no app restart.
Restore-test passed first: ak-pg backup restored cleanly (25 users, 368 tables,
1225 MB) using the corrected RECOVERY.md recipe, with a clean no-leak teardown.
Verified: kubectl kustomize apps/authentik builds with Cluster/ak-pg + exactly
5 prune-protected; bash -n bootstrap.sh + e2e-test.sh OK; kubernetes/seed/ fully
removed; no stale seed file-path references remain.
Left intentionally: barman/hetzner ExternalSecrets still read OpenBao keys under
seed/authentik/... (vault paths, not repo paths).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The host-network switch (#406) broke both proxy members: the
reverse-proxy runs as a non-root user, and on the host netns the default
ip_unprivileged_port_start=1024 denies binding :443 — the container
crash-loops with 'listen tcp :443: bind: permission denied'. In bridge
mode Docker set the sysctl to 0 inside the container netns, which is why
the published-port setup never hit this.
Set net.ipv4.ip_unprivileged_port_start=443 persistently on the proxy
hosts (governs v6 binds too, despite the name). The crash-looping
container self-heals within its restart backoff once the sysctl lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stage 1 of folding seed-authentik into the existing dedicated authentik KS — the
last app in the seed-layer retirement, and the highest blast radius (ak-pg backs
the SSO identity provider). Adds prune: disabled, in place, to the 5 CNPG DB
objects under seed/authentik/ so the ownership handoff can never prune them:
- Cluster/ak-pg (HA, 2 instances — pruning deletes the identity DB's data PVCs)
- ObjectStore/ak-pg-store, ScheduledBackup/ak-pg-daily
- ExternalSecret/barman-cloud-creds, ExternalSecret/hetzner-s3-creds
authentik has no data PVCs in seed/ (all state is in ak-pg). Metadata-only; no
CNPG restart. ak-pg has 30 completed barman backups (last ~6.7h) as a backstop,
and a restore-test is running against it now (corrected #404 recipe) before the
stage 2 fold.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every connection of the k8s netbird-ingress peers relays through the
single controller-VPS relay (verified 2026-07-25: 5/5 connected peers
Relayed, zero P2P) — which is why the 2026-07-24 Hetzner DC outage took
the public edge down even though the reverse-proxy cluster runs at a
different provider. Three structural blockers, two fixed here:
- netcup proxy members: run the reverse-proxy container on the host
network instead of bridge+443:443. Behind the bridge the embedded
client's WireGuard/ICE UDP sat on a random Docker-NAT66 port that the
pods' egress policy could never match; on the host network its ICE
host candidates are the un-NATed public addresses, reachable
outbound-only from the pods (no hole punching). proxy-2 serves
v4+v6, proxy-1 v6-only — pods have working egress on both families
(verified). :443 binds as before; the hosts run no firewall
(verified), so exposure is unchanged.
- netbird-ingress-egress CNP: allow pod-to-pod between the ingress
peers (previously absent entirely — even same-cluster peers relayed
via Hetzner) and any-port UDP-capable egress to the three proxy
addresses (the embedded client's port is 51820 by default but not
contractual).
The controller relay remains available as designed fallback. Merging
auto-triggers the deploy-netbird-proxy Woodpecker pipeline (serial: 1,
one member at a time) and Flux applies the CNP.
Verify after rollout: netbird status -d in a netbird-ingress pod should
show 'Connection type: P2P' for the proxy peers and the sibling ingress
peers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the immich seed->apps move (same single-transfer pattern as paperless
#400). Restore-test passed first (im-pg backup restorable: 33316 rows in asset).
- git mv seed/immich/{cluster,objectstore,scheduled-backup,barman-creds-
externalsecret,hetzner-creds-externalsecret,pvcs}.yaml -> apps/immich/, add to
apps/immich/kustomization.yaml.
- immich KS: drop 'dependsOn: seed-immich', add
'dependsOn: infrastructure-cnpg-barman', add the CNPG Cluster healthCheckExprs.
- Delete clusters/homelab/seed-immich.yaml and the now-empty seed/immich/.
- e2e-test.sh: drop seed-immich, add immich to the app-KS readiness checks
(immich was previously only covered via seed-immich).
- Fix the last stale seed/immich comment path in
infrastructure/openbao/snapshot-pvc.yaml (seed/immich -> apps/immich).
Safe ownership transfer (nothing pruned): all 7 seed/immich objects carry
prune: disabled (5 CNPG from stage 1 #401, 2 PVCs pre-existing), so deleting
seed-immich skips them in GC and the immich KS adopts them in place. No CNPG
restart, no app restart.
Verified: kubectl kustomize apps/immich builds 17 objects incl. Cluster/im-pg,
exactly 7 prune-protected; bash -n e2e-test.sh OK; seed/immich fully removed.
Left intentionally: the hetzner-creds ExternalSecret still reads OpenBao key
seed/immich/... (vault path, not a repo path).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The restore-test drill in RECOVERY.md told you to omit spec.plugins from the
throwaway cluster ('NO spec.plugins / isWALArchiver here'). With the plugin-based
barman this homelab now uses (barmancloud.cnpg.io ObjectStore + the
barman-cloud.cloudnative-pg.io plugin), that is invalid: the instance manager
can't load the plugin it needs to replay WAL during restore and dies at startup
with 'Unknown plugin: barman-cloud.cloudnative-pg.io' — the cluster never becomes
healthy. A real cold-rebuild DR following this drill would have failed the same
way. Found + fixed by running the drill against im-pg on 2026-07-25 (restore
succeeded once the plugin was declared; 33316 rows in 'asset', 584 MB DB).
- Declare the barman plugin in spec.plugins with isWALArchiver: false — loaded so
it can READ WAL for the restore, but never archives (stays read-only against
the backup path). Update the intro prose to match.
- Verification: pg_stat_user_tables n_live_tup reads 0 right after a restore
(stats not gathered), so that view is not a data check — add pg_database_size
and note counting a real table instead.
- Teardown: the old 'reclaim the released PV afterwards' was too vague and leaks
the Retain PV's RBD image (issue #262). Give the explicit recipe: capture the
PV name, delete the cluster, then patch the released PV to Delete so the CSI
reclaims the RBD image.
The in-place cold-rebuild section (which keeps the live cluster's existing
spec.plugins isWALArchiver: true) was already correct and is unchanged.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the 14aa6415 review.
- pick-model.sh: the picker relied on `--allowedTools ""` to be tool-less, but an
empirical check showed an empty allowlist does NOT block tools (the Write tool
still ran) -- it is treated as "unset" and the default toolset stays live. Add
`--permission-mode plan`, which IS verified to block Write while still producing
the selection JSON. --allowedTools "" + the Read deny-list remain as additional
layers. Corrected the comments in pick-model.sh and the picker settings.json
that claimed the empty allowlist made it tool-less.
- .woodpecker/ci-scripts-test.yml: new pull_request-triggered pipeline (homelab
only) that runs scripts/ci/test-model-selection.sh, so a PR weakening the
injection gate fails checks immediately instead of only on the next scheduled
e2e cron (closes the coverage-timing gap the review flagged).
- Pin a recorded real /v1/models response (scripts/ci/testdata/v1-models-sample.json)
and assert fetch-models.sh maps it correctly (opus effort=true, haiku
effort=false), documenting the verified response shape.
Deferred with rationale: hard-failing a pipeline when the degraded-schema warning
fires -- the warning already reaches the job log (curl/jq stderr is no longer
suppressed), and hard-failing on a benign API schema addition would break all
model selection; graceful degradation to name+date is the deliberate choice.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A 24-min WAN outage to netbird.renner.uno on 2026-07-24 exceeded the
10-min ephemeral-peer lifetime, so management deregistered all three
routing peers. The client treats 'peer is not registered' as
unrecoverable and parks in NeedsLogin; that fails readiness but not the
'--check live' liveness probe, so the pods sat Running-but-NotReady for
~20 h until they were deleted by hand.
- liveness probe now fails on the dead auth states (NeedsLogin /
LoginFailed / SessionExpired) and only on those: a restart re-runs the
entrypoint, which re-enrolls with NB_SETUP_KEY. Transient
Disconnected/Connecting states still retry silently, so a management
outage does not kill the working WireGuard data plane.
- new PrometheusRule: warning when the DaemonSet runs degraded >10m,
critical when 0 peers are Ready >5m.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pr-review triage agent runs with an unscoped Write tool and its cwd contains
./.pr-review (-> $D), so a prompt-injected diff could make it overwrite
./.pr-review/models.txt and then name a smuggled token as its `model`. Validation
re-read models.txt FROM DISK after the triage ran, so the catalog-membership
check was defeated by the agent's own writeback -- letting an attacker force a
weaker real model, or a syntactically-valid fake id that makes the review's
`claude --model <fake>` fail and silently suppress the review entirely.
Validate against $MODELS_TABLE -- the catalog captured in a shell variable BEFORE
the triage runs -- instead of re-reading the file the triage can overwrite. This
mirrors pick-model.sh, which already validates against its in-memory table before
its (tool-less) picker runs. The triage still reads models.txt for its own pick;
tampering only misleads that pick, which is then rejected against the pristine
snapshot -> CLI default (so the review still runs, it can't be suppressed).
Verified: a token present only in a tampered on-disk file but absent from the
pre-triage snapshot is now rejected (it was accepted before). Added a regression
test to scripts/ci/test-model-selection.sh, and fixed the header comment to note
the complex-review override as another path to the CLI default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the a7890e0a review.
- pr-review.yml: strip REVIEW_TOKEN from the valid-model.sh call too (it runs a
script from $HOMELAB, branch-supplied for a homelab PR), and guard it with
`|| true` so a missing script can't kill the step -- restoring the stated "a
run never dies on selection" invariant. For a complex/security-sensitive
review, ignore the (injectable) triage pick and run on the CLI default
(strongest configured) model.
- valid-model.sh: reject a leading dash at the shape check (making the header
accurate) so an option-like value is refused even without a catalog.
- fetch-models.sh: --retry 2 --retry-connrefused so a transient blip doesn't
drop the run to the default; add a FETCH_MODELS_BODY_FILE test seam.
- pick-model.sh: pass --settings (new ci/agents/model-picker/settings.json,
Read deny-list) as belt-and-suspenders behind --allowedTools "".
- test-model-selection.sh: add fetch-models.sh mapping tests (well-formed body
-> non-null capability columns; degraded body -> loud warning) via the seam,
and wire the whole suite into .woodpecker/e2e.yml so it runs in CI.
- Condense the duplicated model-selection comment in the four fixed jobs to a
one-line pointer to pick-model.sh.
- PR description: state the cost / no-fixed-ceiling property change.
Deferred with rationale (in the PR thread): selected-model in the four crons'
posted artifacts (already logged per run), pagination past limit=1000 (API max;
has_more warns), and unit-testing pick-model.sh's JSON extraction (its output
always passes through the tested valid-model.sh gate, and the picker prompt is
developer-controlled, not attacker input).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 1 of folding seed-immich into the existing dedicated immich KS (same
pattern as paperless #399/#400). Adds prune: disabled, in place, to the 5 CNPG
DB objects under seed/immich/ so the ownership handoff can never prune them:
- Cluster/im-pg (pruning deletes its data PVCs)
- ObjectStore/im-pg-store, ScheduledBackup/im-pg-daily
- ExternalSecret/barman-cloud-creds, ExternalSecret/hetzner-s3-creds
The immich-library (300Gi) + ml-cache PVCs already carry the annotation.
Metadata-only; no CNPG restart. im-pg has 30 completed barman backups (last
~2.5h) as a backstop.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the remaining pr-review items.
- Extract the untrusted-model validation out of the pipeline YAML into
scripts/ci/valid-model.sh (single safe-token + catalog-membership gate) and
reuse it from both pick-model.sh and pr-review.yml, so the security-critical
logic lives in one tested place. Add scripts/ci/test-model-selection.sh, an
offline fixture test (no network / claude / Woodpecker) that pins the
behaviour: valid ids pass; hallucinated, empty, space-, newline-, leading-
dash-, and metachar-injected values are rejected; empty catalog rejects all.
This is the regression test for the flag-injection fix.
- pr-review triage cost: hand the triage the COMPLETE changed-file list
(files.txt) plus a diff capped at 4000 lines (diff.triage.patch) instead of
the full up-to-20k review diff, so classifying a PR on the default model stays
cheap. stats.txt still carries the true totals; triage-prompt.md updated.
- Doc nits: note the new helpers are POSIX `sh` (not `bash` like the older
scripts/ci/*.sh), and that fetch-models.sh's json mode is a debug affordance.
Deferred with rationale (also in the PR thread):
- Model id in the four cron jobs' posted artifacts: selection (including the
fallback-to-default) is already logged per run in the CI output, the
operational surface for this; threading it through four different posted
formats is a separate observability follow-up.
- --settings sandbox on the picker: it is granted no tools (--allowedTools ""),
so there is nothing for a Read-deny sandbox to constrain -- the other agents'
sandboxes exist to scope tools they DO have.
- Pagination past limit=1000: that is the API's max page size; >1000 models is
not a real scenario, and the has_more warning turns even that into a visible
log line rather than silent truncation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the paperless seed->apps move. Unlike woodpecker this is a single
ownership transfer into the app's existing dedicated KS.
- git mv seed/paperless/{cluster,objectstore,scheduled-backup,barman-creds-
externalsecret,hetzner-creds-externalsecret,pvcs}.yaml -> apps/paperless/, and
add them to apps/paperless/kustomization.yaml.
- paperless KS: drop 'dependsOn: seed-paperless', add
'dependsOn: infrastructure-cnpg-barman', add the CNPG Cluster healthCheckExprs
(moved off seed-paperless), refresh the stale 'seed phase' comment.
- Delete clusters/homelab/seed-paperless.yaml and the now-empty
kubernetes/seed/paperless/.
- e2e-test.sh: drop the seed-paperless readiness check (the paperless app KS is
already checked).
- Fix two stale file-path comments that pointed at the old location:
apps/woodpecker/objectstore.yaml and infrastructure/openbao/snapshot-pvc.yaml
(seed/paperless -> apps/paperless).
Safe ownership transfer (nothing pruned): all 9 seed/paperless objects carry
prune: disabled (5 CNPG objects from stage 1 #399, 4 PVCs pre-existing), so
deleting seed-paperless skips them in GC and the paperless KS adopts them in
place. No CNPG restart (Cluster is not a Deployment), no app restart.
Verified: kubectl kustomize apps/paperless builds 40 objects incl. Cluster/
pprl-pg, exactly 9 prune-protected; bash -n e2e-test.sh OK; seed/paperless fully
removed; no stale file-path refs remain (only the dated migration log + the
OpenBao KV path pending the separate hetzner-creds decoupling).
Left intentionally: the hetzner-creds ExternalSecret still reads OpenBao key
seed/paperless/hetzner-creds (a vault path, not a repo path).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 1 of folding seed-paperless into the existing dedicated paperless KS.
Adds kustomize.toolkit.fluxcd.io/prune: disabled, in place, to the five CNPG DB
objects under seed/paperless/ so the ownership handoff can never prune them:
- Cluster/pprl-pg (HA, 2 instances — pruning deletes its data PVCs)
- ObjectStore/pprl-pg-store, ScheduledBackup/pprl-pg-daily
- ExternalSecret/barman-cloud-creds, ExternalSecret/hetzner-s3-creds
The 4 data/media/consume/redis PVCs already carry the annotation. Landing it via
the existing seed-paperless KS guarantees it's live before the move. Metadata-
only; no spec change, so CNPG and the running pprl-pg instances are untouched.
Unlike woodpecker this is a single ownership transfer (seed-paperless ->
paperless KS, both already dedicated), so no app-object commonAnnotations / CI
restart is involved. pprl-pg has healthy barman backups (last ~2.7h) as a
backstop.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pr-review triage agent reads the attacker-controlled PR diff and writes
triage.json; its Write tool is not path-scoped, so a prompt-injected diff can
make it emit any `model` string. The catalog validation used
`grep -qxF -- "$MODEL"`, and grep -F treats a MULTI-LINE operand as
newline-separated patterns: a value like
"claude-sonnet-5\n--dangerously-skip-permissions" matches on its first line,
passes validation, and -- via the unquoted `${MODEL:+--model $MODEL}` expansion
-- word-splits into an extra argv entry, injecting a CLI flag into the review
agent's own `claude` invocation (defeating its --allowedTools allowlist and Read
sandbox, with CLAUDE_CODE_OAUTH_TOKEN in that process's env).
Removing map_model() (which returned one of three literal ids) is what opened
this; restore the invariant with a single-token guard before the expansion:
`case "$MODEL" in ""|*[!A-Za-z0-9._-]*) MODEL= ;; esac`. Verified it turns the
newline/space/hallucinated payloads into a plain `claude -p` (CLI default) while
still passing real ids. Mirrored the guard in pick-model.sh as defence in depth
(its grep -oE already keeps CHOICE single-line, so it was not exploitable).
Also from the review:
- pr-review.yml: correct the HOMELAB comment -- the fetch-models.sh call is
env -u REVIEW_TOKEN'd, so CLAUDE_CODE_OAUTH_TOKEN (not REVIEW_TOKEN) is what
branch-supplied shell there can see.
- pick-model.sh / triage-prompt.md: stop asking the picker to weigh "cost" the
catalog does not expose; reason from the visible capability signals (context /
max_output / effort|thinking|code_exec) as the scale proxy instead.
- fetch-models.sh: drop 2>/dev/null on curl so its -S error reaches the log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final stage of the seed-woodpecker -> dedicated woodpecker KS move. The
commonAnnotations prune:disabled guard on apps/woodpecker/ has done its job: the
ownership handoff completed (verified — wp-pg + all app objects adopted by the
woodpecker KS, RESTARTS=0, nothing pruned), so the app objects no longer need
blanket prune protection and should be normally prunable by their owning KS again.
Removes only the commonAnnotations block. The 4 CNPG DB objects keep their
per-file prune: disabled (permanent data-guard on Cluster/wp-pg etc.) — verified
kubectl kustomize apps/woodpecker now annotates exactly those 4, not the 13 app
objects.
Rolls woodpecker-server + agent once more as the annotation leaves their pod
templates — the same accepted CI restart as stage 1.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- pick-model.sh: stop swallowing fetch-models.sh and claude stderr so the real
cause of a fallback (no token / HTTP 401 / parse error / auth) reaches the job
log instead of just "no catalog available"; time-bound the selection call with
a self-probing `timeout` (the wrapper is skipped unless a probe confirms the
image's timeout takes coreutils "SECS CMD" syntax, so it can never silently
route every run to the default); strip PUSH_TOKEN and REVIEW_TOKEN as well --
issue-fix binds its bot PAT a second time as PUSH_TOKEN, which survived the
original `env -u GITEA_TOKEN`; `grep -qxF --`.
- fetch-models.sh: warn loudly when every model returns a null context window,
so a /v1/models schema change surfaces as a message rather than silently
reducing both selectors to name+date guessing.
- pr-review.yml: `env -u REVIEW_TOKEN` on the fetch-models.sh call (it needs only
CLAUDE_CODE_OAUTH_TOKEN), matching the agent calls below it; `grep -qxF --`.
Deferred with rationale: hard-bounding the triage's diff input (kept as prompt
guidance -- a hard cap risks its security-sensitivity judgment) and an offline
fallback-path test harness (tracked as a follow-up).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- fetch-models.sh: request ?limit=1000 so older/cheaper models are not dropped
by the API's 20-item default page; warn on has_more. Add a --table mode so
the catalog format lives in ONE place (pick-model.sh and pr-review.yml now
consume it instead of each re-deriving the jq table). Clean up the HTTP-000
double-print and surface jq parse errors instead of hiding them.
- pick-model.sh: run the picker with GITEA_TOKEN stripped from its env and
without acceptEdits (it is granted no tools); consume fetch-models.sh --table.
- pr-review.yml: print model `CLI default` instead of an empty backtick pair on
the fallback path; document that the fetch-models.sh call runs branch-supplied
shell (not a new exposure -- Woodpecker already runs the pipeline YAML from
the PR branch).
- triage-prompt.md: read stats + the changed-file list first and pull the full
diff only when needed, so the triage (now on a capable default model) does not
read a 20k-line diff just to classify a trivial PR.
Verified against the live catalog: all 11 currently-listed ids run under
`claude --model`, and the capability fields (context/effort/thinking/...) are
present in the OAuth /v1/models response -- so the reviewer's blocking concern
(a listed-but-CLI-unusable id) has no live trigger, and the fallback-to-default
path still covers it if that ever changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restructures woodpecker into its own Flux Kustomization instead of splitting its
app across the catch-all apps KS and its DB across seed-woodpecker.
- Add clusters/homelab/woodpecker.yaml: dedicated KS over apps/woodpecker with
the infrastructure-cnpg-barman dep + CNPG Cluster healthCheckExprs (moved off
seed-woodpecker) so the CNPG concerns live here, not in the shared apps KS.
- Remove woodpecker/ from apps/kustomization.yaml (catch-all apps KS no longer
owns the woodpecker app objects).
- git mv seed/woodpecker/{cluster,objectstore,scheduled-backup,
hetzner-creds-externalsecret}.yaml + MIGRATION.md -> apps/woodpecker/, and add
the four DB manifests to apps/woodpecker/kustomization.yaml.
- Delete clusters/homelab/seed-woodpecker.yaml and the now-empty
kubernetes/seed/woodpecker/.
- bootstrap.sh: add a wait_kustomization woodpecker entry (it used to be covered
by 'wait apps'); e2e-test.sh: add woodpecker to the KS readiness checks.
Safe dual ownership transfer (nothing pruned): every object carries prune:
disabled from stage 1 (#394) — the app objects via commonAnnotations, the DB
objects per-file — so when the apps KS drops woodpecker/ and seed-woodpecker is
deleted, both GC sweeps skip these live objects and the new woodpecker KS adopts
them in place. No CNPG restart (Cluster is not a Deployment); no app restart
(manifests unchanged, only the owning KS label flips).
Verified: kubectl kustomize apps/woodpecker builds 17 objects (13 app + 4 DB
incl. wp-pg Cluster), all prune-protected; bash -n bootstrap.sh + e2e-test.sh
OK; wp-pg is the sole CNPG cluster in apps ns and hetzner-s3-creds is used only
by wp-pg-store; seed/woodpecker fully removed.
Stage 3 will drop the transient commonAnnotations (one more accepted app roll).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pr-review, update-audit, security-scan, alert-monitor and issue-fix
pipelines each pinned a dated model id (--model claude-sonnet-5, plus
pr-review's haiku/sonnet/opus map_model allowlist). Those go stale every
time a new model ships and bake in assumptions about which model families
exist, so a newly released model could never be chosen without a code edit.
Replace all of it with selection driven only by what is available right now:
- scripts/ci/fetch-models.sh queries the catalog the CLI itself uses
(Anthropic GET /v1/models, via the existing CLAUDE_CODE_OAUTH_TOKEN and the
required anthropic-beta: oauth-2025-04-20 header) and emits each model with
its objective capability signals -- context, max_output, effort, thinking,
code_exec, release date -- with no id patterns and no family names.
- scripts/ci/pick-model.sh lets a triage agent choose the best-fit id from
that catalog for a one-line task description, weighing every listed model
on its merits. pr-review's existing per-PR triage agent now does the same,
reading the live catalog instead of emitting a fixed haiku/sonnet/opus tier.
No model id -- and no model family -- is baked into any pipeline. A model
released tomorrow, of any family, becomes eligible the day the API lists it.
The bootstrap selection call runs on the CLI default model, and if the catalog
is ever unreachable --model is simply omitted (CLI default) rather than falling
back to a named model, so a run never dies over model selection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Folds the app-object protection into stage 1 now that a woodpecker CI restart is
acceptable. commonAnnotations adds kustomize.toolkit.fluxcd.io/prune: disabled to
every object under apps/woodpecker/ (the woodpecker-ci Namespace, server/agent
Deployments, Service, Ingress, PDB, NetworkPolicy, ResourceQuota/LimitRange,
RBAC, ExternalSecret), so when 'woodpecker/' is later removed from
apps/kustomization.yaml the catch-all apps KS can't prune them mid-handoff.
Trade-off accepted: kustomize propagates commonAnnotations into the two
Deployments' pod templates, so this rolls woodpecker-server + agent once on
apply (and once more when removed in the final stage). Confirmed harmless — the
DB (wp-pg) is a separate CNPG resource, untouched.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 1 of moving woodpecker off seed-woodpecker into a dedicated woodpecker KS.
Adds kustomize.toolkit.fluxcd.io/prune: disabled, in place, to the four
data-bearing DB objects in seed/woodpecker/ so the later ownership handoff can
never prune them:
- Cluster/wp-pg (pruning deletes the DB's data PVCs)
- ObjectStore/wp-pg-store (barman backup target)
- ScheduledBackup/wp-pg-daily
- ExternalSecret/hetzner-s3-creds (barman S3 creds)
Landing the annotation via the existing seed-woodpecker KS guarantees it is live
on the objects before any move. Metadata-only; no spec change, so CNPG/the
running wp-pg pod are untouched. The Cluster keeps the annotation permanently as
a data-guard; the others are incidental to the same handoff.
The woodpecker *app* objects (server/agent/Namespace/etc., currently under the
catch-all apps KS) are protected separately in the next stage — they need
per-object top-level annotations rather than commonAnnotations, which would
propagate into the Deployments' pod templates and needlessly restart live CI.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the follow-up review on #390: three more comments still described the
removed influxdb-seed Job, some in the present tense (contradicting the note the
prior commit added).
- helmrelease.yaml adminUser comment: drop the 'sibling influxdb-seed Job reads
the secret / deletes the bucket' description; state the admin token comes from
the influxdb-bootstrap Secret (OpenBao via ESO).
- helmrelease.yaml resource/timeout rationale: reword 'after a partial seed' and
'batches from the seed Job' to the seed-independent conditions (large dataset /
large write batches) that still justify the 6Gi limit + 60s write timeout.
- ha-token-job.yaml: drop 'Same wait pattern as the seed Job's wait-for-influxd';
describe the two-stage wait directly.
Verified: kubectl kustomize apps/influxdb builds; the only remaining 'seed'
strings are accurate migration-rationale comments and the OpenBao KV path.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Without the suffix the app resolves devices only via its 2-minute mDNS
cache; the authoritative-DNS leg of MdnsConnectCallback is disabled and
the presence sweep falsely marks online always-on devices missing
(5 devices confirmed online but flagged, 2026-07-24).
Fixes#389
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses the doc-only suggestion on #390: explain in the new influxdb KS that
dropping seed-influxdb's infrastructure-cnpg-barman dependency is intentional
(copy-paste boilerplate; InfluxDB has no CNPG cluster, only a ceph-block PVC).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Step 2 of the InfluxDB seed->apps handoff. #388 already landed
prune: disabled on the live HelmRelease/HelmRepository/ExternalSecret, so
retiring seed-influxdb here cannot trigger a helm uninstall — GC skips the
annotated objects and the new influxdb KS adopts them in place.
- git mv kubernetes/seed/influxdb/ -> kubernetes/apps/influxdb/ (annotations
from #388 travel with the files).
- Add kubernetes/clusters/homelab/influxdb.yaml (KS 'influxdb', path
apps/influxdb, deps infrastructure/crds/ceph). Delete seed-influxdb.yaml.
Review fixes from the closed atomic PR (#387):
- bootstrap.sh: wait on 'influxdb' instead of the deleted 'seed-influxdb'
(would otherwise spin the full 600s timeout on cold bootstrap).
- e2e-test.sh: swap the seed-influxdb readiness check for 'influxdb'.
- Fix stale comment paths kubernetes/seed/influxdb/... -> apps/influxdb/... in
network-policies/home-assistant-ns.yaml and apps/monitoring/
helmrelease-prometheus.yaml.
- Trim the helmrelease.yaml comment that described the removed seed-job.yaml,
and the ha-token-job.yaml 'at cutover moves to apps/influxdb' note (this is
that cutover).
Left intentionally: externalsecret.yaml key 'seed/influxdb/bootstrap' is an
OpenBao KV path, not a repo path — moving it needs a vault-side secret
migration, out of scope here.
Verified: kubectl kustomize apps/influxdb builds all objects with the right
prune/force annotations; bash -n passes on bootstrap.sh + e2e-test.sh; no
remaining seed-influxdb references in the repo (only the OpenBao KV path).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Step 1 of a two-step seed->apps handoff for InfluxDB (the move itself follows
in a separate PR). Adds kustomize.toolkit.fluxcd.io/prune: disabled to the
HelmRelease, HelmRepository, and ExternalSecret *in place* under
kubernetes/seed/influxdb/, with no move and no KS changes.
Why split: doing the annotate + move + delete-old-KS in one commit races the
new KS's first reconcile (which stamps the annotation onto the live objects)
against the old KS's deletion-GC. If GC wins, it prunes the un-annotated
HelmRelease -> helm uninstall -> the running InfluxDB StatefulSet is torn down.
Landing the annotation via the existing seed-influxdb KS first guarantees it is
live on the objects before any prune can run, per Flux's documented pattern.
No behavioural change: only a kustomize metadata annotation is added, so
helm-controller performs no upgrade and the server is untouched. After merge,
verify the annotation is live on all three objects, then land the move PR.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second step of dissolving the transitional 'seed' layer (after
music-assistant). Move the audiobookshelf config/metadata/books PVCs from
seed-audiobookshelf into the app's own Kustomization and retire the seed KS.
- git mv kubernetes/seed/audiobookshelf/pvcs.yaml -> kubernetes/apps/audiobookshelf/
and add it to the app kustomization.
- Drop 'dependsOn: seed-audiobookshelf' from the audiobookshelf KS (it now owns
the 3 PVCs directly; ceph dep already present, no CNPG involved).
- Delete kubernetes/clusters/homelab/seed-audiobookshelf.yaml and the now-empty
kubernetes/seed/audiobookshelf/.
- Remove seed-audiobookshelf from the e2e-test.sh Kustomization readiness checks
(it no longer exists post-merge) — same fix the review flagged on the
music-assistant PR, done proactively here.
Safe ownership transfer: all 3 PVCs carry kustomize.toolkit.fluxcd.io/prune:
disabled, so Flux GC skips them when seed-audiobookshelf is pruned; the
audiobookshelf KS adopts them on reconcile. Verified: PVCs Bound + prune-disabled
live; kubectl kustomize apps/audiobookshelf builds all 3.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review on #385.
Blocking: e2e-test.sh asserted kustomization/seed-music-assistant Ready, but
this PR deletes that KS — the test would fail post-merge with 'not found'.
Remove it from the checked list (line 110).
Also add a clarifying note in MIGRATION-sops-openbao.md that
seed-music-assistant has since been retired (the 2026-07-14 log line is left
factually intact as a historical record).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First step of dissolving the transitional 'seed' layer: move the
music-assistant-data PVC from seed-music-assistant into the app's own
Kustomization and retire the seed KS.
- git mv kubernetes/seed/music-assistant/pvcs.yaml -> kubernetes/apps/music-assistant/
and add it to the app kustomization.
- Drop 'dependsOn: seed-music-assistant' from the music-assistant KS (it now
owns the PVC directly; ceph dep already present, no CNPG involved).
- Delete kubernetes/clusters/homelab/seed-music-assistant.yaml and the now-empty
kubernetes/seed/music-assistant/.
Safe ownership transfer: the PVC already carries
kustomize.toolkit.fluxcd.io/prune: disabled, so Flux GC skips it even when the
seed-music-assistant KS is pruned — the object is never torn down, and the
music-assistant KS adopts it on next reconcile. Verified: PVC is Bound and
prune-disabled live; kubectl kustomize apps/music-assistant builds with the PVC.
Pilot for the per-app seed->apps relocation; remaining apps (audiobookshelf,
influxdb, woodpecker, paperless, immich, authentik) follow the same pattern.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The monolithic `Kustomization/seed` was split into per-app `seed-<app>`
KSes; it was left suspended + prune-disabled only so its inventory could
be pruned by a follow-up commit. That inventory is now empty (ownership
fully migrated to the per-app KSes), so the object owns nothing and can
be removed cleanly — flux-system (prune: true) will garbage-collect the
now-absent Kustomization/seed, and its own prune: false + empty inventory
means zero live resources are torn down.
Also drop kubernetes/seed/kustomization.yaml, the aggregator that only the
monolith referenced (path ./kubernetes/seed); it was already stale (missing
woodpecker). The per-app KSes target ./kubernetes/seed/<app> subdirs and are
unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gitea 1.21.4 logs a deprecation warning for [server] LFS_CONTENT_PATH
(superseded by [lfs] PATH) and recommends DISABLE_QUERY_AUTH_TOKEN
(default from 1.23). Both surface as startup notices on the admin
dashboard.
Add the replacements as GITEA__ env overrides so they are tracked in
git. environment-to-ini can only add keys, so the stale
[server] LFS_CONTENT_PATH was also removed from the on-PVC app.ini by
hand (same path value -> existing data/lfs objects untouched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The push trigger matched any branch, so pushing this feature branch ran the
refresh pipeline from the branch: it built and pushed ci-claude:2.1.218 and
opened a bump PR (#379) prematurely. Restrict the push event to branch: master
so only master ever builds/pushes images or opens bump PRs; cron and manual
are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The bump credential is a claude-bot PAT scoped write:repository only (a
dedicated Gitea user needs admin to create, which claude-bot lacks; the
scoped-down PAT is the same least-privilege at the token level). Token minted
and stored as the ci_claude_bump_bot_token Woodpecker secret (events:
push/cron/manual). Commit author set to claude-bot accordingly. Only the cron
registration remains, to be done after this merges to master.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ci-claude image installed a fixed, pinned Claude Code version and every
consumer pipeline pinned a stale dated image tag. Two problems:
- The npm install layer changed on every build (install-time mtimes + the
npm cache), so it never deduped in the Gitea registry — the single largest
orphan pattern behind #336 (~260 MB orphaned per no-op rebuild).
- A frozen version silently rots: old Claude Code releases eventually stop
working against the API.
Containerfile: default to @latest (npm auto-selects the musl native binary on
this alpine base) and make the install layer byte-stable — clean the npm cache
and normalize install-time mtimes on the installed tree and the directories the
RUN touches. Two independent --no-cache builds now produce a byte-identical npm
layer, so a rebuild of the same version dedups instead of orphaning a copy.
New ci-claude-refresh.yml owns the image lifecycle: weekly (and on-demand) it
resolves the latest published version, and only when it differs from what the
consumers pin does it build ci-claude:<version> (+:latest) and open a PR
rewiring the 5 consumer pipelines. Version tags make "did it change?" a
race-free string compare and keep the cron from opening no-op PRs. Nothing
auto-merges. build-ci-images.yml no longer builds ci-claude so the two never
fight over tags.
Requires owner setup before it does anything (documented in the pipeline
header): a dedicated ci-claude-bump-bot identity (write:repository only) as the
ci_claude_bump_bot_token secret, and a cron named "ci-claude-refresh".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review blocker: my previous filename fix used bash substring syntax
${TALOS_SCHEMATIC:0:12}, but the apply step runs under busybox ash
(Alpine /bin/sh) which errors 'Bad substitution' — the repo already uses
cut for this exact reason. Replaced with $(echo "$TALOS_SCHEMATIC" |
cut -c1-12). The HCL substr() in talos_image.tf is unaffected (not shell).
Also finishes the comment sweep the prior commit claimed but didn't: 4
more stale 'eth1' host-NIC references in the matter/otbr/music-assistant
NAD headers are now ens19. Grep confirms zero eth1 left in the v30iot
NAD/deployment files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
The step exports REVIEW_TOKEN (write:repository PAT) for its own curl/gitea_post
calls, but that also placed it in the environment of the claude triage+review
agents, which process attacker-controlled PR content — a prompt-injected diff
could read /proc/self/environ and exfiltrate it into the public report. The
agents never call Gitea (the shell does, via $AUTH), so run them under
'env -u REVIEW_TOKEN'. Defence in depth alongside the settings.json /proc deny.
Caught by the review bot; matches the fix applied to the other fleet repos.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review follow-up (optional): the shelly/matter/otbr NAD header comments
still described the old eth1 probe-order naming, contradicting this PR's
own root-cause writeup. Updated to the stable ens19 predictable name and
dropped the stale k8s-edge/probe-order narration.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Review blocker: the Proxmox image-exists check derived TALOS_FILENAME
from version only, so bumping the schematic left the cached filename
unchanged — a future VM (re)provision would find the stale old-schematic
image and skip the download, silently reintroducing net.ifnames=0. Embed
the schematic hash in the filename (pipeline + opentofu/talos_image.tf,
kept in sync) so a schematic bump forces a fresh download.
Also reconciles stale comments the rename left behind: the MA reconcile
docstring (hostNetwork-era eth1 explanation), cluster.yaml kubelet nodeIP
(eth1->ens19), and the matter NAD's k8s-edge claim (edge isn't migrated
and isn't a current node; the pod is pinned off it anyway).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Root cause of the IoT NIC name shuffle: the nocloud platform driver
unconditionally injects net.ifnames=0, disabling systemd predictable
naming, so eth0/1/2 followed driver-probe order and renamed across
reboots (a k8s-3 reboot put the v30iot NIC on eth2 and mgmt on eth1,
silently mis-targeting the v6-disable sysctl onto mgmt and attaching
macvlan children to the Server VLAN). LinkAliasConfig cannot fix this
(it only sets a cosmetic IFLA_IFALIAS, invisible to sysctl/CNI).
Fix: re-enable predictable naming via a new Image Factory schematic
(extraKernelArgs: -net.ifnames=0, net.ifnames=1). Because the cluster
boots sd-boot/UKI, the cmdline is baked into the image — machine.install
extraKernelArgs is silently ignored — so this MUST be delivered as a
schematic + talosctl upgrade, not a config patch. The busPath-derived
names are topology-stable and identical across the three identical VMs:
0000:06:13.0 (v30iot) -> ens19
0000:06:12.0 (mgmt) -> ens18
0000:02:00.0 (ceph) -> enp2s0np1
Re-keys everything that referenced the volatile eth1 onto the stable
ens19: the four v30iot macvlan NADs (matter/otbr/shelly/music-assistant)
and the per-node v6-disable sysctl. Bumps the schematic id in the deploy
pipeline. Applied live via rolling talosctl upgrade (one node at a time,
full Ceph recovery between each); verified ens19 identical on all three,
v30iot NIC addressless v4+v6, macvlan children retain their own v6.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
The earlier edit that removed the stale suspend-comment block also dropped the
'spec:' line, leaving suspend/schedule/jobTemplate nested under metadata: — an
invalid CronJob that would be rejected by the API server (kustomize build does
not validate against the k8s schema, so it passed). Re-add spec: and sync the
top-of-file doc comment to the reconciled infrastructure/claude#setup-token
path. Caught by the review bot on this PR; verified with kubectl --dry-run=server.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gate zigbee2mqtt.home.renner.uno behind the same passkey-step-up Authentik
proxy-outpost pattern used for esphome/shelly-manager/vscode. The Z2M frontend
has no auth of its own (anyone on :8080 controls the whole Zigbee network), so
Traefik only ever routes the hostname to a dedicated outpost that validates the
session before forwarding.
New:
- apps/home-assistant/zigbee2mqtt-gate-service.yaml — ExternalName mirror to
ak-outpost-zigbee2mqtt-gate in the authentik ns (:9000).
- apps/home-assistant/zigbee2mqtt-ingress.yaml — zigbee2mqtt.home.renner.uno →
the outpost; cert-manager ingress-shim issues zigbee2mqtt-tls.
- apps/authentik/zigbee2mqtt-gate-egress.yaml — CNP letting the outpost egress
to the zigbee2mqtt pod on :8080.
Edited:
- apps/authentik/blueprints-configmap.yaml — 5-entry blueprint block (group /
proxyprovider / application / policybinding / outpost) named zigbee2mqtt-gate;
internal_host carries the explicit :8080 (Service listens on 8080, unlike
esphome's :80). Authentik spawns the ak-outpost-zigbee2mqtt-gate Deployment
+Service from the outpost entry.
- infrastructure/network-policies/home-assistant-ns.yaml — fromEndpoints rule
admitting the zigbee2mqtt-gate outpost to :8080 (shared 8080 with vscode/
shelly-manager; the fromEndpoints selector + the authentik-side egress scope
it to the zigbee2mqtt pod).
- the two kustomizations registering the new files.
DNS (k8s-gateway watches Ingress) and TLS (cert-manager) are automatic. Access
limited to the "Zigbee2MQTT Managers" group (k.renner). Verified: kustomize
builds, dry-run OK, blueprint YAML parses with the new block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pin both CronJobs to the built secret-rotator image that carries woodpecker-secret
mode (sha256:828bf486…), flip suspend:false, and reconcile the claude-oauth
BAO path to homelab/infrastructure/claude#setup-token (where the OAuth value is
actually seeded) — with the matching policy update (applied out-of-band).
Bootstraps + policies/roles are already applied and the org secrets are live, so
the rotators now maintain them on schedule: review-bot mints+syncs review_bot_token
quarterly; claude-oauth syncs claude_code_oauth_token daily.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The coordinator_backup.json approach did NOT make herdsman adopt the existing
network. herdsman's zstack manager compares the CONFIG network params against
the adapter's NV and refuses on mismatch; a backup only prevents accidental
re-commissioning, it does not supply the network to an already-commissioned
adapter. So Z2M sat on defaults (PAN 0x1a62 / channel 11) vs the real network
(PAN 0x8b2b / channel 25) and correctly refused — no re-pair, no damage.
Set the real network params explicitly (values are the adapter's own, read from
herdsman's own mismatch log, cross-checked against the OpenBao backup which
matches exactly: key fe0b..2d, pan 8b2b, ext fc7a735d08c15476, ch 25):
advanced:
channel: 25
pan_id: 0x8B2B
ext_pan_id: [0xfc,0x7a,0x73,0x5d,0x08,0xc1,0x54,0x76] # natural order
network_key: "!secret network_key"
Byte order pinned against zigbee-herdsman v10.6.1 source (determineStrategy):
ext_pan_id is reversed only when WRITING to NV, but compared/logged
non-reversed, and the network key is never reversed — so both go in natural
order matching the displayed adapter hex.
network_key stays secret: the init container extracts it from the OpenBao
coordinator backup (jq + sed -> natural-order int array) into
/app/data/secret.yaml, referenced by the config as a quoted "!secret" ref.
channel/pan_id/ext_pan_id are non-secret and live in the ConfigMap.
Verified: kustomize builds, deployment dry-run OK, extraction produces the
correct 16-byte array, config schema types valid. The final config<->adapter
match is confirmed at startup, where the coordinator_backup.json guarantees a
safe refusal (never a re-commission) if anything is still off.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Unblocks the node-unaddressing change: MA ran hostNetwork and its
reconcile initContainer fails closed if no 10.2.30.0/24 address exists
on a host interface — removing node host addressing would crash-loop it
on every node it can schedule to (no edge toleration).
MA now attaches net1 via macvlan on v30iot (DHCP-leased), exactly like
matter-server/otbr/shelly-manager: its own IoT address, direct L2
adjacency for Snapcast/SlimProto/Sendspin audio sync (macvlan bridge
mode puts frames straight on the wire — same property hostNetwork gave,
without the node holding an IoT address). The reconcile script is
unchanged in logic: it scans the pod's interfaces for the 10.2.30.0/24
address and now finds net1 (Multus attaches it with its lease before any
container runs) instead of the host eth1. bind_ip stays 0.0.0.0 -> binds
eth0 + net1; publish_ip = net1's leased IP. NAD omits matter's Thread
accept_ra tuning (MA has no Thread); keeps sbr (leased default gateway
would otherwise blackhole cluster-internal replies).
Folded into this PR so the change is atomic: on merge Flux migrates MA
to its own IoT lease before the manual node-unaddressing rollout removes
the host addresses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Root cause of Z2M refusing to start, verified against Z2M's own validator: the
MQTT password was written as a bare YAML tag `password: !secret mqtt_password`,
and Z2M 2.x does NOT register a `!secret` YAML tag — js-yaml throws
"unknown scalar tag !<!secret>", which surfaces as the generic
"configuration is invalid (use yaml-validator)" message.
Z2M's secret mechanism is parseValueRef (settings.ts read()): it matches the
regex `!<file> <key>` against the STRING VALUE of mqtt.user/password/server,
advanced.network_key, frontend.auth_token. So the reference must be a QUOTED
string, not a tag:
password: "!secret mqtt_password"
Reverted the interim env-override attempt: ZIGBEE2MQTT_CONFIG_* is applied only
in write()/onboarding, not on a plain read of an existing config, so it did NOT
set mqtt.password (confirmed: mqtt.password was empty). The quoted-!secret +
init-rendered /app/data/secret.yaml path is the one Z2M actually resolves on
read.
Verified end-to-end on the REAL rendered ConfigMap via `settings.validate()` in
the running image: VALIDATE_ERRORS: [], mqtt.password resolves from secret.yaml,
31 devices parsed.
Post-merge: the PVC still holds the pre-fix seeded configuration.yaml (init only
seeds when absent), so it must be cleared once so it re-seeds from this fixed
ConfigMap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review follow-ups (non-blocking):
- The disable_ipv6 sysctl keys on the literal 'eth1' while the interface
is selected by busPath; a rename would no-op the sysctl SILENTLY and
leave the host v6-addressed on IoT. Document the dependency and that
the rollout must verify eth1 is addressless per node.
- shelly-manager-deployment comment said 'master: eth2' — the NAD uses
eth1. Last lingering eth2-means-v30iot reference, corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Review on PR #363:
- Suspend both CronJobs at merge (review-bot quarterly, claude-oauth daily):
the pinned image is still :latest and predates woodpecker-secret mode, so an
un-suspended merge would fail on schedule with 'unknown mode'. The image-pin
follow-up flips them on. (suggestion 2)
- GITEA_VERIFY_REPO: note shelly-manager verified private 2026-07-23 (unauth
404 + Woodpecker SCM PRIVATE=yes) and the degradation-if-public risk, so the
pull-only gate's assumption is documented. (suggestion 1)
- make-before-break header: caveat that the 'sync-fail keeps working' property
holds for one cycle; a persistent sync failure must be fixed before the next
quarterly run reaps the still-referenced token. (suggestion 3)
Left as-is per the review: the woodpecker.rs defensive else-arms (mirror
bao.rs) and no ureq-mocking test (repo convention).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Security + routing. The k8s nodes held a DHCP-leased v4 address and an
RA-derived v6 address on the IoT VLAN — a network that must be treated
as compromised. That gave every IoT device an L3-reachable path to the
node (kubelet, SSH-equivalent apid, any host service) and, as a side
effect, installed a connected 10.2.30.0/24 route so node replies to
IoT-subnet clients bypassed the UDM entirely — breaking every UDM-DNATed
flow (the hairpin NAT: un-translated SYN-ACK -> client kernel RST,
verified live 2026-07-23).
The NIC (busPath 0000:06:13.0) stays up as the macvlan PARENT for the
v30iot pod attachments (matter-server, otbr, shelly-manager), whose
children do their own DHCP in their own netns — host addressing was
never needed for that. dhcp: false drops the v4 address+route;
sysctl net.ipv6.conf.eth1.disable_ipv6=1 drops v6 including link-local,
so the host has no addressable presence on the L2 at all. The old
comment's claimed purpose (Cilium L2 announce for an IoT LB pool) does
not exist — no L2AnnouncementPolicy, BGP-only pool.
RESIDUAL (not addressed here): the NIC remains a macvlan parent, so the
node kernel still processes frames from the IoT L2. Full node isolation
from IoT would require relocating how those three pods reach IoT devices
— a separate, larger change.
Same suppression pattern the repo already uses (edge-host 10-vlan40,
HA macvlan /32) rather than policy routing (Talos 1.12 has no
RoutingRuleConfig; that is 1.13+).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Live cutover fix. Two changes:
1. Remove the top-level `permit_join: false` from the Z2M config. It was
REMOVED in Zigbee2MQTT 2.0 — a config containing it fails schema validation
and Z2M refuses to start (serves the failure page). Joining is runtime-only
now (default off; toggled via frontend or the bridge/request/permit_join
topic), which is what we want anyway. Verified the other 2.x-sensitive keys
(homeassistant.enabled, frontend.enabled, mqtt.*) against the current docs —
they were already correct.
2. Flip replicas 0 -> 1. The cutover is committed: ZHA is disabled, the
coordinator-backup Secret is populated in OpenBao, and Z2M is now the live
Zigbee stack. Flux owns this field, so it must be 1 in git (a manual
`kubectl scale` gets reverted on the next reconcile). Rollback = set 0 and
re-enable ZHA.
Note: the running PVC still holds the pre-fix seeded configuration.yaml; the
init container only seeds when absent, so after merge that stale file must be
cleared once (delete it + restart the pod) so it re-seeds from this fixed
ConfigMap. Tracked as the immediate post-merge step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a woodpecker-secret mode to secret-rotator that upserts an OpenBao-held
value into a Woodpecker ORG-level CI secret (create if absent, else patch
value+events), and wire two rotators that use it — moving review_bot_token and
claude_code_oauth_token off hand-loading into the rotation toolchain and onto
org-level secrets every repo inherits.
secret-rotator:
- woodpecker.rs: org-id lookup, existence probe (404=absent), create/patch an
org secret via a dedicated woodpecker-bot bearer token. Never logs the value.
- main.rs: woodpecker-secret mode reads a value from BAO_PATH#BAO_FIELD (errors
if missing/empty, so a blank never overwrites a working secret) and the
woodpecker-bot token from the bootstrap, then upserts the org secret with the
configured events (must include pull_request for the PR-review pipeline).
rotators (kubernetes/infrastructure/openbao/rotation/):
- review-bot.yaml: one CronJob, init step mints review-bot's PAT (gitea-token
mode, pull-verified against a private repo it collaborates on; push not
verified — it's a read collaborator that posts reviews via the API), main step
syncs it to org secret review_bot_token (events cron,manual,pull_request).
- claude-oauth.yaml: sync-only (the OAuth token is interactive setup-token, not
mintable) — daily push of an owner-seeded OpenBao value to org secret
claude_code_oauth_token.
- policies/rotator-{review-bot,claude-oauth}.hcl: least-privilege OpenBao
policies + documented hand-seeded bootstraps (review-bot password,
woodpecker-bot token, seeded OAuth/token fields).
netpol: rotator-{review-bot,claude-oauth}-egress (openbao-ns) +
gitea-allow-review-bot-rotator + a new woodpecker-allow-rotators, so the
rotators reach gitea:3000 / woodpecker:8000 through apps default-deny.
Deploy prerequisites (can't be done from git): pin the image @sha256 once a
build including woodpecker-secret mode lands; apply the two OpenBao policies/
roles; seed the bootstraps. After the org secrets exist, drop homelab's
repo-level claude_code_oauth_token / review_bot_token copies.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Woodpecker runs its ${...} variable substitution over the ENTIRE pipeline YAML
including comments, so the literal dollar-brace in the triggering comment
(added when documenting the dropped manual path) failed to parse — 'unable to
parse variable name' — erroring every pull_request pipeline before any step
ran. Reword the comment to avoid the glyph; the live homelab pipeline was down
since #362 merged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The triage + review agents run with REVIEW_TOKEN and CLAUDE_CODE_OAUTH_TOKEN in
their environment, process attacker-controlled PR title/body/diff, and their
output is posted verbatim as a public PR comment. Without a Read sandbox a
prompt-injected diff could read /proc/self/environ (the tokens) or .git/config
(Woodpecker's clone credential) and get it published. Every other agent
pipeline already denies this via --settings; pr-review had none.
Add ci/agents/pr-review/settings.json (mirrors issue-fix/update-audit, gitea
#197) denying Read on /proc,/root,/home,/etc,.git, and pass --settings
$PROMPTS/settings.json to both claude invocations (resolves alongside the
prompts, so it works for the repos that fetch prompts from the homelab clone).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correct the cutover mechanism before it ever runs. The prior approach hand-set
advanced.pan_id/ext_pan_id/network_key in the Z2M config from the ZHA values.
That is the risky path: the extended PAN ID byte order is REVERSED between ZHA
(zigpy) and Z2M (zigbee-herdsman), so loading the ZHA value as-is makes herdsman
see a mismatch and FORM A NEW network — dropping every device (the explicit
non-goal). Mixing a manual ext_pan_id with a backup also conflicts (Z2M ignores
the reversed value).
Instead restore an Open Coordinator Backup, the shared zigpy<->herdsman
interchange format, which pins byte orders in-spec and handles the reversal on
import:
- New zigbee2mqtt-coordinator-backup ExternalSecret (OpenBao) holds the full
backup JSON (contains the network key -> OpenBao only, never git), replacing
zigbee2mqtt-network-externalsecret.yaml.
- Init container seeds /app/data/coordinator_backup.json on first run only
(guarded on database.db too, so Z2M's own state is never overwritten), and
now renders only the mqtt password into secret.yaml.
- Config drops advanced.pan_id/ext_pan_id/network_key/channel entirely; the
network comes from the restored backup.
Runbook updated: Step 1 exports the Open Coordinator Backup from zigbee.db via
zigpy (non-disruptive — no second radio session); Step 2 stores it in OpenBao;
added an explicit device re-interview step (devices stay joined but Z2M's
database starts empty), and a PVC-clear note in rollback.
Verified live: coordinator has pan 0x8b2b, ext_pan_id fc7a735d08c15476,
channel 25; zigpy emits a valid open-coordinator-backup for it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a pull_request event Woodpecker's built-in clone already checks the repo out
at the PR head in the workspace, so the agents read that directly for context.
Restores full-context review (the prior commit over-corrected to diff-only)
while keeping the manual, token-bearing clone gone. Prompts point at the
working-directory checkout instead of ./work/target.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clone existed only to hand review_bot_token to git, which persisted it in
work/target/.git/config — a file the review agent (Read/Glob/Grep, output
posted publicly) could read. Nothing needed the checkout that the API-fetched
diff + metadata don't already provide, so drop the clone entirely: no token on
disk, no work/target, no cleanup. The PR + diff are still read over the API
with ephemeral curl Authorization headers (never written to disk).
Prompts updated to match: the agents review the diff alone (no ./work/target),
and are told to flag, not guess, when a judgement needs code they can't see.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pull the live ZHA device registry (31 devices + coordinator) and pre-seed each
device's friendly_name in the Z2M config, keyed by IEEE. Devices then come up as
"EG Küche Tischlampe Oben" instead of "0x001788010856bb17" the moment Z2M
re-interviews them on the adopted network — which is what keeps the HA-side
entity reconciliation small and mechanical. Two devices that were both bare
"Deckenlampe" in ZHA are disambiguated by area (Z2M requires unique friendly
names): EG Galerie (Hue LTC001) and DG Lager (LUMI N1 switch).
Add zigbee2mqtt-entity-reconciliation.md: an audit of exactly which ZHA entities
the git-tracked config references, so the cutover knows what must be re-pointed.
Key result — the HA-side change is FAR smaller than "rewire everything":
- 216 ZHA entities exist, but only 16 distinct entity_ids are referenced in
the repo (200 are unreferenced diagnostics/config entities).
- 14 of the 16 are physical-device entities, all mapping cleanly into the
31-device inventory (no orphans).
- The remaining 2 are the `light.texas_instruments_coordinator_*` entities —
now identified as ZHA *groups* on the coordinator (not physical devices),
accounting for 14 references. These don't come from Z2M device discovery and
must be re-created as Z2M groups and re-pointed by hand.
Scope note captured in the doc: only git-tracked YAML/JSON was audited; live
.storage Lovelace dashboards and UI-created automations/helpers must be
reconciled separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the credential-in-URL clone (which git wrote into
work/target/.git/config, readable by the review agent whose output is posted
publicly) with a tokenless clone that supplies auth via an HTTP header through
GIT_CONFIG_* env. The persisted remote is tokenless, so no repo file holds the
PAT — fixing the leak at the source instead of deleting .git after the fact.
Supersedes the previous rm -rf .git stopgap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clone embedded REVIEW_TOKEN in the URL, which git persists in
work/target/.git/config. The review agent runs next cd'd into the workspace
with Read/Glob/Grep and its report is posted verbatim as a public PR comment,
so the write:repository PAT was readable and could be exfiltrated via prompt
injection from the reviewed PR (caught by the review bot on PR #180). The
agent needs only the working tree, so drop .git entirely after checkout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lay down Zigbee2MQTT (Z2M) in the home-assistant namespace as the eventual
replacement for the ZHA integration, wired to drive the SAME coordinator ZHA
uses today (SMLIGHT SLZB-06 / TI CC2652, tcp://slzb-zigbee.renner.uno:6638).
Devices are joined to the network on the coordinator chip, so if Z2M comes up
with the existing network parameters (pan_id, ext_pan_id, network_key, channel)
zigbee-herdsman ADOPTS that network instead of forming a new one — every device
stays joined, no re-pairing.
Ships INERT and safe to merge:
- Deployment is replicas: 0. Reconciling this PR starts nothing.
- The ZHA network params come from the `zigbee2mqtt-network` ExternalSecret
(OpenBao), populated only at cutover. Until then the ExternalSecret is
NotReady and the pod cannot start, so it can never accidentally form a new
network (which would drop every device).
The cutover is a manual, documented, reversible procedure — see
zigbee2mqtt-migration-runbook.md.
Contents:
- zigbee2mqtt-{pvc,config,deployment,service}.yaml — the app. Config template
is seeded to the PVC once; secrets (mqtt password + ZHA network params) are
injected via /app/data/secret.yaml (rendered each start by an init
container) and referenced with Z2M `!secret`, keeping the network key out
of git and out of the ConfigMap.
- zigbee2mqtt-network-externalsecret.yaml — OpenBao-backed network params,
populated at cutover from a ZHA network backup.
- New `zigbee2mqtt` Mosquitto broker user: bootstrap Job
(mosquitto-bootstrap.yaml) + passwd/ACL/env (mosquitto-deployment.yaml),
scoped to zigbee2mqtt/# plus write homeassistant/# for HA MQTT discovery.
- zigbee2mqtt-migration-runbook.md — the cutover + rollback procedure.
No new egress CiliumNetworkPolicy is needed: the namespace-wide default-deny
policy already grants same-namespace (mosquitto), DNS, and world (the IoT-VLAN
coordinator) to every pod.
Follow-up (separate homeassistant-repo PR): re-point automations/dashboards
from ZHA entity ids to the new Z2M MQTT-discovery entities.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Woodpecker exposes --var values to YAML ${...} substitution and when-filters
only, not as a step's shell environment, so the documented
'--var PIPELINE=pr-review --var PR=<n>' re-review read an unset shell var and
silently no-opped with a green build (caught by the review bot on PR #49).
Trigger on pull_request only; re-review by pushing a commit (new head sha
clears the dedup marker). Resolve the PR from CI_COMMIT_PULL_REQUEST alone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch the PR review bot from a homelab-hosted hourly cron that polls four
repos over the Gitea API to a per-repo, event-driven pipeline that reviews
only the PR that triggered it (CI_REPO + CI_COMMIT_PULL_REQUEST). Fires on
pull_request, which Woodpecker raises on both PR-open and push-to-open-PR.
Drops the cross-repo discovery loop and MAX_PRS cap; keeps the sha-dedup
marker (now guarding pipeline retries), Haiku triage, model selection, and
never-approve posture. Resolves the agent prompts from the local checkout or
a shallow homelab clone so the identical file drops into every reviewed repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner directive: .k is never reintroduced, not even as a temporary
garbage-collection scope. The leftover k. records on the UDM are out of
external-dns's view and get deleted once via the UDM UI at cutover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Review finding: domain filters gate deletion as well as creation, so
dropping the suffix would orphan the retired k. records instead of
removing them. The filter entry stays until sync has deleted them all
(no resource carries a k. host, so desired state for the suffix is
empty), then gets dropped — the comment says exactly when. Also fixes
the stale excludeDomains mention and the '13 networks' miscount in the
NAT header comments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
*.k.renner.uno was a temporary migration namespace and is banned (owner
directive 2026-07-14, documented in openbao/ingress.yaml) — it was never
the live naming. Removes the .k. hosts/SANs from ical-proxy, loki,
grafana and the mosquitto cert, renames syslog.k.renner.uno to
syslog.renner.uno (cert + external-dns annotation; any syslog sender
still configured with the k. name must be repointed), fixes the stale
comments, updates e2e-test.sh probes to the real names, and drops
k.renner.uno from the external-dns domainFilters so leftover k. records
get garbage-collected by policy: sync.
The migration script's historical docstring reference stays — it
documents what the migration did.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Correction: only domain filters apply to both desired and current
state. The label filter is source-side only, so the registry-owned UDM
records for newly labeled resources stay visible and policy: sync
deletes them itself — the cutover is fully automatic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
The dns.renner.uno/public=true label on an Ingress marks it as publicly
registered (Cloudflare wildcard -> NetBird proxy); external-dns-unifi
now runs with labelFilter dns.renner.uno/public!=true and skips those
resources entirely. Visibility is a property of the app resource itself
— there is no central name list to keep in sync with the proxy.
Also removes the dead .k.renner.uno hosts from the public apps'
Ingresses (their TLS certs reissue without those SANs) and splits
photos.renner.uno onto its own unlabeled Ingress: the label filter is
per-resource and that name stays LAN-registered. Remaining .k. names on
LAN-only apps (loki, ical-proxy, syslog) are untouched here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Review finding: on (40) Server, (42) Ingress and (49) k8s Ceph the
cluster nodes sit on the same L2 as the client, so a DNATed flow's
reply bypasses the UDM conntrack (asymmetric) — and clients there are
the nodes themselves, which already resolve the public names in-cluster.
Their idx slots stay reserved. Also records the restapi provider in the
lock file and trims trailing blank lines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Review finding: the domain filter applies to current state as well, so
pre-existing LAN records for newly excluded names are orphaned on the
UDM and must be removed once by hand as part of the cutover.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Per review: no family-specific behavior. Every rule is the same
ADDRESS_AND_PORT specific-address shape (endpoint x port x network,
6 rules per network), matching the pre-existing hand-made rules'
structure. Drops the tofu-owned firewall groups and the v4-only
FIREWALL_GROUPS path entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
No UniFi Terraform provider models v2 NAT translation fields
(ubiquiti-community 0.55 none, filipowm #104 open, resnickio shell-only),
so the rules go through Mastercard/restapi against the controller's
/proxy/network/v2/api/site/default/nat endpoint, same API key as the
unifi provider.
Rule shapes follow the behaviorally verified matrix on Network 10.5.62:
v4 uses FIREWALL_GROUPS (tofu-owned address+port groups); v6 must use
ADDRESS_AND_PORT specific-address rules (4 per network) because v6
group-based DNAT silently drops the address match and rewrites all
traffic on the listed ports. rule_index is static per network to keep
indices collision-free and stable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
Now that woodpecker runs on postgres (not the previously-pinned sqlite), the
"never surprise-bump a live sqlite DB" hazard is gone. Bump server + agent
3.13.0 -> 3.16.0 (a reviewed minor jump: no breaking config changes for our
kubernetes+gitea+postgres setup, no risky migrations per the changelog; direct
3.13->3.16 is supported, migrations run on startup).
Add Flux image automation (image-automation/woodpecker.yaml) at range 3.x, so
minor + patch releases auto-update; only the 4.0 major stays a reviewed bump.
Server and agent share the same range/tag-filter and both carry an $imagepolicy
marker, so they bump in lockstep (gRPC version parity is required).
Pre-upgrade CNPG base backup taken (wp-pg-preupgrade-316) as the rollback path,
since migrations are forward-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 3/3, the finale of the sqlite->CNPG migration. woodpecker-server is now
stateless (DB in wp-pg Postgres, verified end-to-end incl. a live pipeline run),
so convert the StatefulSet to a Deployment and remove the volumeClaimTemplates +
/var/lib/woodpecker mount. strategy: Recreate keeps a single control-plane
instance (no two servers on one DB during a roll).
This deletes the last use of the data-woodpecker-server-0 PVC, whose ext4
CSI-snapshot backup failed permanently ("structure needs cleaning"). Once the
PVC itself is removed, Velero's apps-daily stops trying to snapshot it —
resolving #295 and #324. The agent reaches the server via the standalone
`woodpecker` Service (label selector), so a Deployment pod is reachable
unchanged.
Closes#295Closes#324
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 2/3 of the sqlite->CNPG migration. Points woodpecker-server at the
wp-pg Postgres cluster (creds from the CNPG-generated wp-pg-app secret via a
lib/pq keyword DSN). The sqlite data was migrated one-time via pgloader
(schema-first/data-only, per seed/woodpecker/MIGRATION.md); parity verified
(all 18 tables match, booleans correct, sequences reset).
The sqlite PVC + volumeMount are intentionally KEPT for now (rollback = revert
these env vars) — they're dropped in stage 3 (StatefulSet->Deployment) after a
soak. Image stays v3.13.0 (no version bump during the migration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A phone that caches the LAN answer for a public app on wifi (10.2.99.86 /
fd99::86) and then roams to cellular blackholes on it: RFC1918 and ULA give
no RST and no ICMP, so the app hangs until TCP timeout instead of failing
fast. That is why restarting the app does not always help — the OS resolver
still hands back the unroutable address.
Excluding these eight names from the UniFi provider makes LAN fall through to
the public *.renner.uno wildcard, so every network returns the same answer and
a stale cache stays correct. LAN traffic still stays local: the UDM hairpins
the proxy addresses back to 10.2.99.86 / fd99::86 (Policy Table -> NAT).
Excluded by record name rather than by label because these Ingresses are
multi-host — immich.k.renner.uno and immich.renner.uno share one object, and
*.renner.uno cannot cover a three-label name, so a label filter would make the
.k. name unresolvable on LAN.
policy: sync, so this deletes 16 UDM records (8 names x A/AAAA) on reconcile.
The source of truth for the list is the NetBird proxy's tunnelled domains, not
the cluster; it must be updated when an app is published or unpublished.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
woodpecker-server runs sqlite on a ceph-block PVC (data-woodpecker-server-0)
whose Velero CSI-snapshot backup fails permanently (ext4 "structure needs
cleaning" — the source fs is error-flagged and journal replay can't clear it).
Rather than repeatedly fsck a fragile sqlite, move woodpecker's DB onto CNPG
postgres like authentik/immich/monitoring/paperless already do — giving proper
WAL-based barman backups and letting the PVC be removed entirely, which
resolves #295/#324 by deletion.
This PR (stage 1 of 3) provisions only the empty backend — no woodpecker
impact:
- kubernetes/seed/woodpecker/: wp-pg Cluster (1 instance, initdb woodpecker),
ObjectStore -> Hetzner S3 (s3://renner-uno-k8s-cnpg/wp-pg), ScheduledBackup,
and a hetzner-s3-creds ExternalSecret (reuses the shared paperless S3 path to
avoid seeding a new OpenBao key; follow-up noted inline to give it its own).
- seed-woodpecker Flux Kustomization (dependsOn infrastructure + cnpg-barman,
CNPG health-gated), auto-synced from clusters/homelab/.
- MIGRATION.md: the one-time schema-first / data-only pgloader runbook and the
staged cutover (PR 2 = server.yaml -> postgres; PR 3 = drop PVC, STS->Deploy).
sqlite PVC is untouched; rollback is reverting env vars until PR 3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The infrastructure Kustomization used wait: true with no healthChecks. With
wait: true Flux ignores healthChecks and blocks the Kustomization's Ready on
EVERY resource it applies (hundreds). Because ~28 kustomizations dependsOn
infrastructure, one crashlooping workload froze the entire GitOps apply chain:
a crowdsec-agent crashloop held all of GitOps hostage for hours — merges didn't
go live — even though crowdsec is a non-critical app nothing depends on for
startup. The 15m health-check timeout looped every 10m, so infrastructure never
reported Ready and every dependent stayed blocked.
Switch to wait: false + explicit healthChecks covering only the operators and
providers dependents actually consume: cilium (CNI), cert-manager (TLS Secrets),
external-secrets + openbao (the ESO backend ExternalSecrets resolve against),
rook-ceph-operator (storage), cloudnative-pg (DB operator for seed jobs).
Everything else still applies — it just no longer gates readiness, so a
non-critical crashloop can't cascade across the cluster. This mirrors the
pattern infrastructure-ceph-cluster already uses (healthCheckExprs for Rook).
Verified live: infrastructure reconciles Ready on the 6 checks (wait=false),
and all dependent kustomizations stayed Ready.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds proxy-2.renner.uno (netcup, Arch) as a second proxy-only cluster member
so proxy.renner.uno can move off netbird.renner.uno entirely. The controller
host shared one IP and port with the reverse proxy, separated only by TLS SNI,
which made destination NAT for LAN hairpin impossible without also capturing
NetBird control-plane traffic.
netbird-proxy listens on :443 only (NB_PROXY_ADDRESS=:443) and has no HTTP
listener, so once proxy.renner.uno stopped resolving to the controller host,
port 80 went unanswered for every off-LAN client. Adds a minimal nginx
redirector publishing :80 on every member to restore the 301.
Deploying that sidecar exposed two defects in the role:
- Any docker-compose.yml change notified the force-recreate handler, rebuilding
the proxy container even for edits that do not touch the proxy service.
`docker compose up -d` already recreates only changed services, so the notify
is dropped there; proxy.env keeps it because compose cannot detect env_file
content changes.
- The play restarted every cluster member in parallel, taking the whole proxy
cluster down at once. Now serial: 1.
Verified: both members serve :443 (200) and :80 (301); a repeat run reports
changed=0 on both with no handler firing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review. k.renner/homelab is public, so the pull verification passes for any
valid token — push is the only load-bearing check. It already defaulted to true
in the binary, but pinning it in the manifest documents the safety invariant at
the call site so it can't silently degrade if the default ever changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the Flux git credential — the dedicated `flux` gitea account — into the
rotation fleet, so it can't silently die again (the cause of the recent GitOps
outage). Quarterly, via the secret-rotator binary's gitea-token mode.
Flow: mint a new PAT (basic auth, the flux password bootstrap) -> VERIFY it can
pull AND push (Flux image-automation writes back) -> store in OpenBao -> ESO
merges it into the flux-system secret -> source-controller reads it fresh. A
token that fails verification is deleted and the run aborts without storing, and
the reap keeps the two newest tokens so the previous one stays valid through the
ESO propagation window — Flux is never stranded.
Reachability: gitea via the in-cluster Service gitea.apps.svc:3000 — the public
gitea.renner.uno name hairpins off the ingress LB from a pod (same as
pve.renner.uno). Needs both netpol halves (gitea-allow-flux-rotator in apps +
rotator-gitea-flux-egress in openbao). Added flux-system to the openbao
ClusterSecretStore's allowed namespaces so the ExternalSecret can sync there.
The ExternalSecret uses creationPolicy: Merge, so an ESO failure leaves the
last-good credential in place rather than blanking Flux's secret.
Tested live end-to-end: job minted flux-gitops-1b8d20, verified pull+push,
stored (OpenBao e26e8bf1 -> 9666320e), kept the previous token (2 tokens, 0
reaped), ESO synced, Flux pull fetched, and image-automation pushed
(Ready=True). OpenBao policy/role + the gitea-flux password bootstrap are
provisioned by hand (see rotator-gitea-flux.hcl).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second review pass (no blocking issues). The important one: the reap deleted
EVERY other prefix token right after store, including the previous run's token —
which is still the live credential in the k8s Secret until ESO syncs the new
value and source-controller re-reads it. Deleting it immediately opens an
availability window (ESO refreshInterval + reconcile lag) where Flux's stored
token has just been revoked. Since this mode exists specifically to avoid
stranding Flux, that window shouldn't exist.
Now the reap keeps the TWO newest tokens (the one just stored + the previous)
and deletes only older ones. tokens_with_prefix returns (id, name) so the
caller can order by recency; the previous token survives until the next
quarterly run, long after ESO has synced.
Also:
- doc-note that PUSH is the load-bearing gate check: for a public verify_repo,
git-upload-pack returns 200 for any valid token, so pull barely gates —
don't set GITEA_REQUIRE_PUSH=false against a public repo.
- fetch tokens with limit=50 (single effective page) and note the count stays
tiny given keep-two reaping, so pagination can't silently miss old tokens.
- tidy the cleanup error message connector.
clippy -D warnings clean; 14 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflect the live 20Gi->40Gi expansion in the manifest as source of truth.
volumeClaimTemplates is immutable on an existing StatefulSet, so landing
this needs a one-time `kubectl delete statefulset gitea -n apps
--cascade=orphan` (pod + PVC survive) for Flux to re-create the STS from
the updated template. Documented inline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-ci-images had a pipeline-level path trigger but no per-step
filters, so any push touching ci/images/** or secret-rotator/** rebuilt
ALL five images. The recent burst of secret-rotator commits therefore
needlessly rebuilt the four ~1.3G ci-* images each time, re-pushing and
overwriting :latest and orphaning ~5G of registry layers per run — a
primary source of the orphaned-blob pileup that filled data-gitea-0.
Add per-step when:path so each build runs only when its own source (or
the pipeline file) changed, or on a manual run. A secret-rotator change
now rebuilds only secret-rotator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
data-gitea-0 hit 100% and gitea started failing writes ("database or
disk is full"). Root cause was NOT tag accumulation — live images are
only 8.7G and well-deduplicated. It was 10.4G (54% of the volume) of
ORPHANED, unreferenced blobs: every time an image tag is overwritten
(the ci-* images and shelly-manager rebuild frequently, shelly-manager
on both cron and push), the previous multi-GB layer set is orphaned.
Gitea's default cleanup_packages GC runs only @midnight with
OLDER_THAN=24h, so a day's worth of orphans piled up faster than it
collected them and filled the disk. A manual GC run reclaimed nothing
because every orphan was <24h old and thus skipped by the age cutoff.
Set the GC to run @every 6h with OLDER_THAN=4h so orphans are reclaimed
same-day. 4h is well beyond any in-flight push, so a live upload is
never collected. Applying this restarts gitea and its RUN_AT_START GC
reclaims the existing ~10G backlog (usage drops to ~9G).
Emergency relief was already applied out-of-band: the PVC was expanded
20Gi -> 40Gi (ceph-block online resize) to clear the outage without a
risky restart at 0 bytes free; documented on the volumeClaimTemplate.
Supersedes #349 (which targeted tag count, the wrong layer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the review on the gitea-token mode.
Blocking: the cleanup only ran on the Ok(false) gate branch. If token_allows
returned Err (a wrong GITEA_VERIFY_REPO, a 5xx) or the OpenBao store failed
after verification passed, the `?` propagated and the just-minted, write-scoped
PAT was orphaned with no delete and no log — contradicting the stated
guarantee, and unlike ha.rs which logs its orphan. Fixed: verify+store now run
in a closure, and ANY error after the mint deletes the token; if the delete
also fails, a CRITICAL line names the live token so it can be revoked by hand.
Also from the review:
- distinguish 401 (minted token rejected — auth, surfaced as an error) from 403
(valid token, insufficient scope — a real gate "no") in token_allows, instead
of collapsing both to Ok(false).
- reject an empty GITEA_SCOPES up front, matching BAO_FIELDS / NB_AUTO_GROUPS.
- add a basic() unit test (gitea.rs had none) pinning the auth-header format;
crate now has 14 tests.
- tighten the doc comments that overstated the guarantee.
clippy -D warnings clean; 14 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rotates a gitea PAT used as a git credential — for Flux's own flux-system
secret. Gitea blocks token-from-token (403), so minting uses BASIC auth with a
dedicated service account's password as the bootstrap, like the Cloudflare and
Proxmox privileged bootstraps.
Make-before-break with a hard safety gate: mint a new token, VERIFY it can both
pull (git-upload-pack) AND push (git-receive-pack, since Flux image-automation
writes back), and only then store it. A token that fails verification is
deleted and the rotation aborts WITHOUT storing — so a bad rotation can never
strand Flux the way a revoked token did. Old rotated tokens (same name prefix)
are reaped after, non-fatally.
The push check is required by default but env-toggleable (GITEA_REQUIRE_PUSH)
for read-only consumers. Scopes, verify-repo, name prefix and field names are
all env-driven.
Validated against the real gitea API on throwaway tokens (zero blast radius,
live flux token untouched):
- write token -> pull 200 + push 200 -> gate PASS
- read-only token -> pull 200 + push 403 -> gate REJECTS it (correctly refuses
to store a token that cannot push)
- list-by-prefix + delete (reap) works
clippy -D warnings clean; 13 tests pass.
No rotator uses it yet — the ExternalSecret + CronJob follow once CI publishes
the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mTLS fix in #343 failed to apply: chart 0.24.0's agent/bouncer/appsec
Certificate templates emit `secretTemplate:` unconditionally while its
child blocks are guarded, so with nothing configured it rendered as null
and cert-manager's webhook rejected it (spec.secretTemplate must be of
type object). The helm upgrade failed and rolled back, so agents stayed
on the fragile register-based path.
Set tls.certManager.secretTemplate.labels so secretTemplate renders as a
valid object. Verified via kubectl apply --dry-run=server against the
live cert-manager webhook: all 4 Certificates now accepted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The crowdsec-agent DaemonSet CrashLoops whenever the LAPI pod restarts.
Root cause: the LAPI has no persistence (PVCs disabled in 629a303d to
break the Ceph bootstrap circular-dependency), so its sqlite machine-
registration DB is ephemeral. Agents register once (init container) and
cache a password; a wiped LAPI no longer knows it -> 401 "machine not
found" -> CrashLoop -> DaemonSet never Ready -> the crowdsec HelmRelease
upgrade --wait times out -> the whole `infrastructure` Kustomization and
every app that depends on it stays unhealthy.
Enable chart mTLS (tls.enabled + tls.agent.tlsClientAuth, cert-manager
self-signed CA). With client-cert auth the agent no longer runs
`cscli lapi register`; the LAPI auto-recreates each machine record from
the presented cert on connect, so a wiped/restarted LAPI self-heals with
no PVC and no manual re-registration. Verified via `helm template`
against chart 0.24.0: register step gone, 4 certs emitted, zero PVCs.
Supersede the #337 persistence attempt (revert lapi.persistentVolume.
data.enabled to false): it reintroduced the Ceph circular-dependency and
did not heal the agents (fresh volume = empty DB). mTLS makes the
ephemeral DB harmless instead of trying to persist it.
Pin the chart to 0.24.x: the floating ">=0.12.0" silently upgraded to
0.24.0 (changed persistence defaults), part of this incident.
Note in traefik/crowdsec-middleware.yaml that the inert ingress bouncer
now needs https + CA trust before it can be re-enabled.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The repo was renamed gitea_admin/homelab -> k.renner/homelab. Flux's
GitRepository still fetched the old URL, which now 307-redirects, and an
authenticated git fetch does not survive a cross-repo redirect
("authentication required: Verify"). Flux froze at revision 90d7a7c4
(2026-07-21 22:45); nothing merged since was being applied.
Point spec.url at the canonical k.renner/homelab.git so source-controller can
fetch again. Applied live in parallel (kubectl patch) to break the chicken-and-
egg — Flux can't read git to fix the URL that stops it reading git.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The LAST rotator. Replaces the inline python with the secret-rotator binary's
proxmox-token mode. Net -48 lines; inline Python is now fully retired — every
rotator runs the one binary, and the openbao image is left only to the
raft-snapshot job.
Tested live end-to-end via the real binary image — the destructive test that
could not be zero-blast-radius, since it rotates the real monitoring token:
- job Complete: DELETE+recreated pve-exporter@pve!monitoring (privsep=0),
OpenBao pve.yml rotated 529d7f66 -> f7ca2141, document well-formed
- the no-verify TLS path worked against the real in-cluster Service (valid LE
cert, hostname mismatch) — the piece that could only be compile-proven
- PVE token recreated privsep=0 (inherits the user's PVEAuditor ACL)
- ESO synced, Reloader restarted pve-exporter, and it recovered: OpenBao and
k8s token_values match, the new token authenticates (/version 8.4.18), and
pve_up=1.0 for the cluster and all three nodes
The brief metrics gap between delete and Reloader-restart is inherent to
Proxmox having no atomic roll (documented in the header); off-hours quarterly
makes it acceptable.
Flux strips the old python `command` on apply (verified on the earlier ports);
port via Flux, not kubectl apply.
Image digest sha256:97c22bcf…
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the advisory review on the proxmox-token mode.
The code claimed the PVE cert is self-signed. It is not — the in-cluster
Service presents a VALID Let's Encrypt cert (verified live: CN=neptune.renner.uno,
issuer Let's Encrypt). Verification fails only on HOSTNAME mismatch: the job
connects to the pve.apps.svc.cluster.local Service (for 3-node failover) while
the cert is for a node FQDN. Corrected the module header and the insecure_agent
comment to say this accurately, and spelled out the real tradeoff — ureq 3 has
no skip-hostname-only option, so it's disable-verification-with-failover (this,
matching the python's prior CERT_NONE) vs pin-to-one-node-FQDN-with-full-verify
-but-SPOF. Noted the deferred improvement (a custom rustls verifier that
validates the LE chain but skips hostname).
Also from the review:
- log a warning when the DELETE result is an Err, so a re-run needn't
rediscover why a first attempt behaved oddly (still non-fatal — a leftover
token makes the POST fail loud).
- documented why the raw URL-path interpolation is safe (fixed operator config,
never untrusted; percent-encoding @ to %40 would break PVE's userid lookup).
No behaviour change to the rotation itself. clippy -D warnings clean; 13 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the last mode — the one that will replace the pve-exporter python rotator,
the hardest of the fleet. Proxmox has no in-place token roll, so this DELETEs
and recreates the token with the same id, then patches OpenBao with a rendered
pve.yml document.
Two capabilities new to the binary:
- no-verify TLS: a dedicated agent with rustls verification disabled, used
ONLY for the PVE calls. The API is reached via the in-cluster Service
ClusterIP, whose cert never matches *.svc.cluster.local, so webpki
verification can't apply. OpenBao still uses the normal agent.
- a rendered document, not a bare value: pve.yml is emitted byte-identical to
the python template (2-space indent, trailing newline, verify_ssl:false),
unit-tested.
privsep=0 on recreate makes the token inherit pve-exporter@pve's own
permissions, so no separate token ACL is re-applied — verified against the live
cluster earlier (the PVEAuditor ACL is on the user, not the token). The auth
credential is the least-privilege secret-rotator@pve!rotate identity, not root.
DELETE ignores its own outcome (a missing token is a no-op; a real auth/perm
problem surfaces at the POST, where it's fatal — a delete that leaves the token
in place makes the POST fail loud with "already exists"). If the recreate fails
after a successful delete, the store never happens and the error is hard: the
consumer is down until re-run. This destructive window is inherent to Proxmox
and unavoidable without changing the token id each rotation.
Validated against real Proxmox 8.4.18 on a throwaway token (zero blast radius,
never touched the real 'monitoring' token): DELETE 200 -> POST privsep=0 form
-> 36-char value, no trailing newline, and the new token authenticates
(/version 8.4.18). render_pve_yml is byte-exact unit-tested. clippy -D warnings
clean; 13 tests pass.
No rotator uses it yet — the manifest port follows once CI publishes the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the inline python NetBird rotator with the secret-rotator binary's
netbird-key mode. Net -30 lines; the login+POST+parse python and the
python:3.13-alpine image go away.
Mints a fresh reusable setup key via POST /api/setup-keys and patches it into
OpenBao. Create-new, not roll-in-place: the old key is left to expire, so
there is no window — it stays valid while the netbird-ingress DaemonSet
re-enrolls its pods on the new one. Nothing is deleted (deliberately — a past
manual delete-filter bug wiped unrelated keys).
Tested live end-to-end via the real binary image:
- job Complete: minted 'ingress-rotated', OpenBao rolled 8a0aa8d4 -> 77f1be0c
- NetBird now has 2 'ingress-rotated' keys (new added, old kept — create-new)
- ESO synced; Reloader restarted the DaemonSet; rolling update (maxUnavailable
1) completed 3/3 with 0 errors
- all 3 pods re-enrolled: the new key shows used_times=3, last_used just now;
pods actively handshaking peers (WireGuard/relay connected)
- old key still valid, untouched
Flux server-side apply strips the old python `command` (verified on the other
ports); a manual `kubectl apply` would not, so port via Flux.
Image digest sha256:49bfddd8…
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the mode that will replace the NetBird python rotator. Mints a fresh
reusable setup key via POST /api/setup-keys and patches it into OpenBao.
Create-new, not roll-in-place: the old key is left to expire (keys are
ephemeral with a bounded expiry), so nothing is deleted — no destructive step,
no window. This deliberately does NOT delete old keys; a past manual cleanup
that filtered keys for deletion wiped unrelated ones (a jq precedence bug), and
that risk has no place in an automated rotator.
NetBird auth is `Authorization: Token <pat>` — NOT Bearer. The key/type/
usage_limit semantics (reusable, unlimited) are fixed; name, expiry, ephemeral,
allow_extra_dns_labels and auto_groups come from the environment. The returned
key is used verbatim (serde_json, no trailing newline). assert_fields_exist
guards the target path. An error that echoes the response redacts the `key`
field so a value can never leak into logs.
Validated against the real NetBird API on a throwaway key (created then
deleted, zero blast radius): TLS verifies with the built-in webpki roots (no
no-verify context needed), POST with Token auth returns a 36-char key with no
trailing newline and the expected settings (reusable/ephemeral/usage_limit 0/
1 auto_group). clippy -D warnings clean; 12 tests pass; the in-cluster pieces
are the same code already proven live by the other modes.
No rotator uses it yet — the manifest port follows once CI publishes the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-ci-images.yml was the last consumer of gitea_token. It used the
admin-owned credential purely as a container-registry login, which meant a
credential that could do anything the admin account could was sitting in a
pipeline that only ever needed to push images.
Replace it with ci_images_token: a PAT scoped to `package: read+write` and
nothing else. Verified — 403 on /user/repos, /admin/users and the issues
API; 200 on the /v2/ registry endpoints kaniko authenticates against.
gitea_token is now unreferenced across every pipeline and can be removed
from Woodpecker and rotated.
The token remains OWNED by gitea_admin because Gitea scopes packages to a
user/org namespace and these images live under gitea_admin/*; a different
account cannot publish there. Moving them to a dedicated org is a
38-reference migration across pipelines and Flux-deployed CronJobs and is
tracked in #330.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the admin-owned gitea_token from every pipeline that runs an LLM
agent, and stops issue-fix borrowing the security-review pipeline's
identity. Each pipeline now holds a credential that can do only what that
pipeline needs.
alert-monitor -> alert_monitor_bot_token (write:issue)
issue-fix -> issue_fix_bot_token (write:issue, write:repository)
update-audit -> update_audit_bot_token (write:issue)
The accounts are write collaborators, but the PAT scope is the effective
cap — verified empirically: the write:issue-only tokens return 200 on the
issues API and 403 on /user/repos and /admin/users.
Also retires two stale rationales in the comments (the gitea_token
workaround and the security_bot_token 403 that motivated it), and changes
issue-fix's git identity from security-bot@renner.uno to
issue-fix-bot@renner.uno so alert fixes no longer appear as
security-review output.
build-ci-images.yml still uses gitea_token for registry/release writes and
needs a write:package token before gitea_token can be retired and rotated;
tracked in #330.
Refs #330. Closes#197.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the two identical Cloudflare python rotators with the secret-rotator
binary's cloudflare-roll mode (added in the prior PR). Net -65 lines; the
duplicated OpenBao-login + PUT-and-parse python and the python:3.13-alpine
runtime image go away.
Both roll their token value in place (PUT /user/tokens/{id}/value) and patch
OpenBao. They differ only in two env values, both parameterised:
cloudflare BOOTSTRAP_TOKENID_FIELD=token-id -> infrastructure/cloudflare
external-dns BOOTSTRAP_TOKENID_FIELD=external-dns-token-id -> apps/external-dns
The admin token comes from the same shared bootstrap, so it stays in one place.
Behaviour preserved: roll-in-place keeps the token id/scope (no window), the
API value is stored verbatim (no trailing newline), and assert_fields_exist
guards the target path. cert-manager needs no reload (reads fresh per DNS-01
challenge); external-dns keeps its Reloader annotation (startup env var).
Tested: the cert-manager rotator was rolled live end-to-end via the real
binary image — OpenBao rolled (d6a4d73b -> 9bd47674), new token verifies active
with the SAME id af708481 (roll-in-place), no trailing newline, ESO synced,
27/27 certs still Ready. external-dns is the identical code path (only the
token-id field + target path differ) and was not rolled to avoid a needless
restart; its PUT/parse was additionally proven against the real Cloudflare API
on a throwaway token in the mode PR.
Note: Flux server-side apply strips the old python `command` when it applies
these (verified against the already-ported Class-A CronJobs, which show
command=<none> live). A manual `kubectl apply` would NOT — it merges and keeps
the stale command — so port these via Flux, not kubectl apply.
Image digest sha256:978fe53c…
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the mode that will replace the two identical Cloudflare python rotators
(cert-manager's token and external-dns's). They differ only in which bootstrap
field holds the token id and which OpenBao path receives the value:
cloudflare bootstrap#token-id -> infrastructure/cloudflare#api-token
external-dns bootstrap#external-dns-token-id -> apps/external-dns#api-token
Both read the admin token from the same shared bootstrap, so it lives in one
place; only the token-id pin differs, keeping the two tokens rolling
independently.
Rolls the token value in place via PUT /user/tokens/{id}/value — same id,
scope and policy, new value — so there is no make-before-break and no
destructive window. assert_fields_exist guards the target path (a typo would
strand the real token), and the API value is used verbatim: a stray newline
would surface as "invalid header field value for Authorization" and crash the
consumer (the bug that hit external-dns once), so the string comes straight
from serde_json with nothing appended.
Handles result as either a bare string or {value: ...} so a future API shape
change fails loud rather than storing "null".
Validated against the real Cloudflare API on a throwaway token (zero blast
radius): PUT with empty body returns success + a bare-string value, new != old
with no trailing newline, new verifies active while the old immediately reads
"Invalid API Token" (confirming roll-in-place has no window). clippy
-D warnings clean; 12 tests pass.
No rotator uses it yet — the manifest port follows once CI publishes the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both agent steps hold Gitea write credentials in their environment
(issue-fix holds a push token too), but neither passed --settings, so
their Read tool was unrestricted — unlike update-audit, which got a
Read sandbox for exactly this reason (gitea issue #197).
That left a complete exfil path: Read /proc/self/environ yields the
token, Write puts it in the agent's output file, and the pipeline posts
that output verbatim as a Gitea issue body / PR body on a PUBLIC repo.
Add the same deny-list used by update-audit (/proc, /root, /home, /etc,
.git) to both agents and wire --settings into each invocation. The repo
checkout, .ci-alert-monitor/ and /tmp/alert-fix/ stay readable, which is
all either agent needs.
This is defense-in-depth only. The root cause — CI pipelines holding an
admin-owned token at all — is tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the inline `head -c N /dev/urandom | base64 | tr -d '\n'` +
`bao kv patch` shell in each of paperless, woodpecker, shelly-manager,
authentik and crowdsec-bouncer-key with the secret-rotator binary's `random`
mode. Net -49 lines; the OpenBao login boilerplate (duplicated in all five)
and the busybox/openbao runtime image go away.
Behaviour preserved exactly:
- byte-compatible output — N raw bytes, STANDARD base64 with padding (RFC
4648 unit-tested); 50 bytes for the Django keys (paperless, authentik),
32 for the rest
- merge-patch, so co-located fields are untouched (WOODPECKER_GITEA_*,
authentik smtp-password)
- shelly-manager's two fields (admin-password, api-key) still land in ONE
patch, no split state
- same k8s-auth roles, SAs, schedule, quarterly timing — the roles already
grant read+patch, which the binary's assert_fields_exist needs
Plus a new safety property the shell lacked: assert_fields_exist refuses to
run if a target field is absent, turning a typo into a loud failure with the
old secret untouched, instead of a silent no-op that adds a stray field.
Image pinned by digest sha256:159e5888… (all three ha-token fixes; the shared
OpenBao code path is the same one `random` uses).
Tested against real OpenBao on a throwaway path+role (zero blast radius, no
live consumer touched):
- happy path: two fields rotated to 44-char values (32-byte base64),
co-located field preserved, multi-field single patch
- fail-safe: a nonexistent target field -> job Failed, clear error, the real
field left unchanged (whole patch aborted, no partial write)
All five pass server-side admission dry-run under the restricted PSA.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New Woodpecker cron pipeline that mirrors the alert-monitor / issue-fix
fleet pattern (cron -> headless `claude` -> Gitea API), but reviews open
pull requests instead of triaging alerts:
- .woodpecker/pr-review.yml: hourly `pr-review` cron. Discovers open PRs
across k.renner/{homelab,shelly-manager,authentik-samba-bridge,
homeassistant}, skips drafts/WIP and PRs already reviewed at their
current head sha (dedup marker), caps at 5/run. For each: clones the PR
head, has a cheap Haiku triage agent pick the model by complexity
(haiku/sonnet/opus; security-sensitive paths force opus), runs the
review with that model, and posts a detailed report as a Gitea PR
review (COMMENT, or REQUEST_CHANGES on a real defect — never APPROVE).
- ci/agents/pr-review/triage-prompt.md: model-selection rubric.
- ci/agents/pr-review/prompt.md: the reviewer's task + report format.
Runs under a dedicated `review-bot` Gitea account (secret
review_bot_token) so reviews are visibly bot-authored and independently
revocable. Static review only — no cluster access; agent tools are
Read/Glob/Grep/Write with no Bash.
Not yet active: needs the review-bot account + token, the review_bot_token
Woodpecker secret, and the `pr-review` cron registered (see file header).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rotates prometheus-token, config-sync-token and shelly-manager-llat via the
secret-rotator binary's ha-token mode (HA mints/revokes LLATs only over its
WebSocket API). All three tested end-to-end against live HA before shipping.
Network: HA is the first IN-CLUSTER rotation target, so it needs BOTH netpol
halves — the openbao default-deny only permits TCP egress to
openbao/kube-dns/apiserver/world (the external-API rotators ride `world`):
- home-assistant-rotator-ingress (home-assistant ns): rotator -> HA :8123
- rotator-home-assistant-egress (openbao ns): the egress complement
Reaches HA on :8123 (HA itself, not the :8124 XFF-anchor proxy) via the
home-assistant-bgp Service ClusterIP; the public name hairpins off the LB.
Both dedicated policies, scoped by the rotator label, not rules on the
namespace-wide default-deny (which would leak to every pod on that port).
Reloader wiring — each token's consumer picks up the new value differently:
- prometheus-token: Prometheus config-reloader auto-reloads; no restart.
- shelly-manager-llat: added to shelly-manager's reload annotation (it reads
the token as a startup env var).
- config-sync-token: the config-sync sidecar reads it at startup and lives in
the HA pod, so the HA Deployment gets a reload annotation. This restarts HA
itself quarterly (04:05, off-hours) — accepted so the full-admin token is
lifetime-bounded like the others.
The binary identifies the outgoing token by its JWT `iss`, not client_name
(the stored names are misleading — prometheus-token is "ServiceMonitor",
config-sync-token was "laptop local"), mints under a unique name (HA rejects a
duplicate), and revokes the old token over a fresh new-token session.
OpenBao policy/role bootstrapped out-of-band (rotator-home-assistant.hcl,
committed for provenance; not Flux-managed).
Tested: prometheus-token and shelly-manager-llat rotated live end-to-end —
OpenBao updated, ESO synced, consumer restarted/reloaded, old token revoked and
gone, new token authenticates (Prometheus scrape 200 with 22k metrics;
shelly-manager 0 HA-auth errors).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Found by running the ha-token rotation end-to-end against live HA. The rotation
was fully correct — new token minted, stored, and the old one revoked — but the
job logged a false "OLD token REMAINS VALID" warning.
Root cause: HA binds a WebSocket session to the refresh token behind its access
token. The rotator authenticated with the OLD token and revoked that same token
over that same session, so HA tore the session down as it processed the delete
— the delete succeeded server-side but the ack could not be read, surfacing as
"connection closed by HA". Verified: after the false-warning run, the old
token's refresh id was already gone from HA.
Fix: after storing the new token, drop the old session and reconnect with the
NEW token to perform the revoke. The session is then not the one being deleted,
so the ack reads cleanly and the report is accurate.
This matters beyond cosmetics: the false warning fired on every successful
rotation, which would train an operator to ignore it — and a genuine revoke
failure would then be indistinguishable noise. With the fix, a warning means a
real un-revoked token.
Verified end-to-end against live HA on a throwaway token: revoke over the new
session returns success=True and the old token is confirmed gone (where the old
flow got "connection closed"). clippy -D warnings clean; 12 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Found by testing ha-token mode against live HA before wiring up the CronJobs.
Home Assistant refuses to mint a second long-lived access token whose
client_name already exists — and make-before-break necessarily mints while the
outgoing token (same base name) is still present. So every real rotation would
fail at the mint step with "Unknown error", exactly as observed:
same-name mint -> FAIL {'code':'unknown_error'}
unique-name mint -> SUCCESS
The earlier ad-hoc test only passed because it minted a NEW name
("Config Sync") rather than rotating an existing one.
Fix: mint under "<base> [<6 hex>]". Correctness does not depend on the name —
the outgoing token is identified by its JWT `iss`, never by name — so a random
tag is free, and it also avoids colliding with an orphan left by a prior failed
run. The failure mode was safe throughout: mint fails before store or revoke,
so the old token stays valid and stored (a no-op, not an outage).
Verified end-to-end against live HA on a throwaway token: unique mint succeeds,
the outgoing token is found by iss, store patches OpenBao, and the old token is
revoked and stops authenticating.
clippy -D warnings clean; 12 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The homelab repo was transferred from owner `gitea_admin` to `k.renner`.
Gitea keeps a 307 redirect for the old path, so git clone/push (which
follow redirects) kept working — but the pipelines' `curl -sf` API reads
do NOT follow redirects, so instead of JSON they received the redirect's
HTML body and the next `jq` died with:
jq: parse error: Invalid numeric literal at line 1, column 3
Under `set -eu` this failed the step before any issue/PR was created,
which is why every alert-monitor / issue-fix / update-audit / security-scan
cron run has been failing and no alert issues were being filed.
- Replace hardcoded `gitea_admin/homelab` -> `k.renner/homelab` in all
API bases, git-push URLs, and cron-add doc comments.
- Add `-L` to the read curls that pipe JSON into jq, so a future rename
degrades gracefully instead of feeding HTML to jq.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses three review findings, all real bugs rather than style:
1. No timeouts anywhere. The Python this replaces passed timeout=30 on every
call; ureq 3's default agent has none and tungstenite has no read timeout,
so a hung endpoint would stall the Job silently until activeDeadlineSeconds
with no useful error — and the cluster demonstrably has hanging paths (the
ingress-LB hairpin). Now: a shared ureq::Agent with timeout_global(30s)
threaded through every OpenBao call, and a 30s read timeout set on the
TcpStream under tungstenite's socket (handling both plain and rustls
streams).
2. RANDOM_BYTES=0 would generate a near-empty secret, store it, and report
success — the same silent-misconfig class assert_fields_exist guards. Now
floored at 16 bytes (128 bits); every real rotator uses 32 or 50.
3. If mint succeeds but the OpenBao store fails, the freshly minted token is
valid for its full lifespan, referenced nowhere, and never auto-revoked
(revocation only targets the OLD stored id). The store error path now logs
the orphan's refresh-token id — recovered from its own JWT — so it can be
revoked by hand, and makes clear the consumer is unaffected because the old
token is still stored and valid.
clippy -D warnings clean; 10 tests pass; scratch image builds (4.92 MB) and
all error paths exit non-zero.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
paperless, authentik, woodpecker, crowdsec-bouncer-key and shelly-manager all
ran the same shell one-liner:
head -c <N> /dev/urandom | base64 | tr -d '\n'
so the port has to be byte-compatible, not merely similar: N raw bytes, the
STANDARD base64 alphabet (+ and /, not URL-safe), WITH = padding, no newline.
Consumers store these verbatim and none filter the character set, so quietly
switching to base64url or dropping padding would change the credential shape
for no reason. The encoder is verified against the RFC 4648 vectors, which are
what coreutils base64 produces, plus a case asserting + and / rather than - and _.
BAO_FIELDS is comma-separated and each field gets its own independent value,
written in ONE merge-patch. That preserves shelly-manager's behaviour, where
admin-password and api-key must land together rather than leaving a window
with one rotated and one stale.
Adds assert_fields_exist as a pre-flight guard. Merge-patch CREATES a field
that isn't present, so a typo in BAO_FIELDS would add `secret-ky`, report
success, and leave the real credential un-rotated — a silent no-op that would
only surface during an incident. Rotation only ever replaces existing fields,
so a missing field is now a hard error that names both what is missing and
what is actually there.
Byte lengths preserved from the originals: 50 for the Django-style keys
(paperless, authentik), 32 for the rest.
Module is `secretgen`, not `gen` — the latter is a reserved keyword in the
2024 edition. 11 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Revoking by client_name would have destroyed live credentials. Checked the
three stored HA tokens against auth/refresh_tokens by decoding each one's JWT
iss claim, and the names are not what they look like:
apps/home-assistant/prometheus-token -> "ServiceMonitor"
apps/home-assistant/config-sync-token -> "laptop local"
apps/home-assistant/shelly-manager-llat-> "Shelly Manager"
Separate, unrelated "Prometheus" and "Config Reconciler" tokens also exist. A
name-matching rotator pointed at prometheus-token would have revoked
"Prometheus" — a live credential belonging to something else — and left the
actual stored token valid. Same for config-sync.
An HA long-lived access token is a JWT whose iss claim IS its own
refresh-token id, so the outgoing token identifies itself exactly. The
rotator now reads that, confirms with HA that the id really is a long-lived
access token, and refuses to proceed if it is not. No name matching anywhere.
The JWT signature is deliberately not verified: this is not authentication,
only reading which id the token claims, and HA remains the authority via the
auth/refresh_tokens confirmation before any revoke.
Adds a small base64url decoder (payloads are unpadded) with unit tests
covering iss extraction, unpadded decoding, non-JWT input, and a JWT with no
iss. 6 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the inline-Python approach before it grows a tenth copy. Today the
fleet carries ~130 lines of Python embedded in YAML strings across 9 manifests,
with auth/kubernetes/login duplicated in all 9 and the call()/merge-patch
helpers in 4 — none of it lintable, testable, or reviewable as code.
The Home Assistant rotator would have been the worst of them: a hand-rolled
WebSocket handshake with frame masking inside a YAML string. HA mints
long-lived access tokens only over its WebSocket API (verified against
2026.7.2 — auth/long_lived_access_token and auth/delete_refresh_token both
exist; the REST API has no equivalent).
Ships as a static musl binary in FROM scratch: 4.85 MB, one file, no libc, no
shell, no package manager. Nothing to patch and nothing for a compromised
rotator to exec. rustls with webpki-roots compiles the trust store into the
binary, so TLS works without a CA bundle on disk — in-cluster traffic is plain
HTTP, but that keeps the door open to take over the Cloudflare and Proxmox
rotators.
ha-token mode is make-before-break, and the ordering is load-bearing: list
existing token ids for the client_name BEFORE minting (so the new token can
never land in the revocation set), mint, write to OpenBao, and only then
revoke. If the OpenBao write fails, the old token is still valid and still
stored, so the run is a no-op rather than an outage.
A failed revoke is deliberately non-fatal but logged loudly: the new token is
already live, and a token meant to be retired that is still valid is a real
finding, not noise.
Verified: static-pie linked and stripped; image contains only the binary
(3.2 MB binary, 4.85 MB image); /bin/sh absent; exits non-zero on usage error;
cargo test passes. The pipeline path trigger now includes secret-rotator/**,
without which source changes would silently not rebuild the image.
No CronJob uses it yet — the Home Assistant rotator needs a network decision
first: the home-assistant namespace is default-deny-ingress, so openbao cannot
reach it, and the public name hairpins off the ingress LB.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous config set IPv6AcceptRA=false on enp1s0.40, which left the
interface with only a link-local address and made edge-1 unreachable over IPv6
on the service VLAN. IPv6 is the primary address family on this network, so
suppressing it to guard against a routing problem in the legacy family was the
wrong trade.
The concern that motivated it was real -- a second default route from the VLAN
40 lease would make return traffic asymmetric across the two VLANs -- but it
only required suppressing the default route and DNS, not addressing itself.
RAs are now accepted with [IPv6AcceptRA] UseGateway=false and UseDNS=false,
mirroring what UseRoutes=false and UseDNS=false already do for DHCPv4. Both
options verified present in systemd 261 (UseGateway added in v250).
Verified after applying to edge-1:
- enp1s0.40 now holds fd40::/64 and 2003:d6:9717:a702::/64 alongside
10.2.40.153/24
- exactly one IPv6 default route, still via enp1s0 (VLAN 30)
- exactly one IPv4 default route, still via 10.2.30.1 on enp1s0
- no DNS servers on the VLAN 40 link
- chronyd and keepalived unaffected
Separately discovered and NOT addressed here: all four NTP servers bind only
0.0.0.0:123, so the service has no IPv6 listener at all -- this predates edge-1
and affects jupiter, saturn and neptune equally. No -4 flag is set and IPv6 is
not disabled in the kernel; the cause is not yet determined.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The VLAN 40 prerequisite is now in place and verified: enp1s0.40 is up with a
DHCP lease of 10.2.40.153/24, and both jupiter (10.2.40.201) and the VIP
(10.2.40.4) answer at ~0.5ms over that interface, confirming layer-2 adjacency
with the existing VRRP members.
NOT APPLIED. The playbook run was blocked pending authorisation, so edge-1
currently has chronyd active and serving but keepalived not installed and the
VIP not held. Applying this flag is what actually joins the VRRP group.
Expected on apply: edge-1 comes up BACKUP at priority 70, below neptune's 80,
so the VIP stays with jupiter. The failure mode to watch for is edge-1 not
seeing the existing advertisements, concluding it is alone, and claiming
10.2.40.4 -- a duplicate holder of the address ~150 clients use. Verify
immediately after applying that edge-1 is BACKUP and does not hold the VIP,
and stop keepalived if it does.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prerequisite for enabling keepalived on edge-1: VRRP needs layer-2 adjacency
with the other NTP members on the service VLAN, and the host has only untagged
VLAN 30. The switch port already permits VLAN 40.
Built to be additive and revertible, because enp1s0 is this headless host's
only path to the network:
- The parent's 20-wired.network is not edited. The VLAN is attached through
a 20-wired.network.d/ drop-in instead, so reverting is deleting three paths.
- The VLAN .network sorts before 20-wired.network and matches explicitly on
Name=. 20-wired.network matches broadly on Type=ether; a VLAN link types as
'vlan' rather than 'ether' (confirmed against networkctl output) so it
should not match anyway, but ordering makes that belt-and-braces.
- UseRoutes, UseDNS, UseNTP and IPv6AcceptRA are all disabled on the VLAN.
Taking a second default route and DNS from the VLAN 40 lease would make
return traffic asymmetric across the two VLANs; untagged VLAN 30 stays the
only default route.
Tasks are tagged vlan40 so they can be applied without running the whole
edge-host role, which would restart the Z-Wave, open3e and alloy services.
Reconfiguring the parent link is what materialises the VLAN child -- a bare
networkctl reload leaves existing links untouched -- so the handler does that
explicitly. That re-runs DHCP on enp1s0 and is the one genuinely risky step;
worth running with console access to hand.
Not yet applied: the apply was blocked pending authorisation, so edge-1 remains
in its original network state with only 20-wired.network and 80-can0.network.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The three PVE hosts currently serve NTP to ~150 LAN clients via chrony with
keepalived holding VIP 10.2.40.4, handed out by UniFi DHCP option 42. Talos
cannot run either daemon, so the planned bare-metal migration removes this
role entirely. edge-1 is brought into the NTP group to take it over.
chrony-ntp-server gains Debian/Arch support. The two distros differ in three
ways that would each have broken on edge-1: config path (/etc/chrony/chrony.conf
vs /etc/chrony.conf), unit name (chrony vs chronyd), and service user (_chrony
vs chrony). Arch also ships no /var/lib/chrony or /var/log/chrony, deferring
them to tmpfiles.d, so the role now creates them explicitly rather than relying
on pacman hook ordering. systemd-timesyncd is stopped where present, since
edge-1 runs it today and it contends with chrony for the clock.
keepalived-ntp is restored -- install-ntp-ha.yml referenced it but the role was
absent from master, so the playbook could not resolve. Recovered from the
homelab-migration-summary worktree with three fixes:
- The health check hardcoded `systemctl is-active chrony`. On Arch that unit
does not exist, so every check would fail, parking the host in permanent
FAULT so it never takes the VIP -- surfacing only at cutover, once the PVE
nodes were already gone. The unit name is now templated.
- Install branches on os_family. Config path and unit name are identical
across distros here, unlike chrony, so nothing else needed changing.
- The sysctl list is now a variable. It unconditionally set
net.ipv4.ip_forward=1, which is correct on the hypervisors but would make
edge-1 a router between IoT VLAN 30 and service VLAN 40 once the tagged
interface exists, bypassing the UDM's inter-VLAN policy. edge_host keeps
only ip_nonlocal_bind; VRRP does not need forwarding.
The two roles have different reach by design. chrony runs everywhere: edge-1
serves from its VLAN 30 address (10.2.30.108) and `allow 10.2.0.0/16` plus
inter-VLAN routing covers every client, so no network reconfiguration is
needed. keepalived is gated on ntp_keepalived_enabled, currently false for
edge-1, because VRRP needs layer-2 adjacency on VLAN 40 and the host has no
enp1s0.40 yet -- the switch port permits the VLAN but /etc/systemd/network/
has no matching .netdev/.network pair.
edge-1's keepalived priority is 70, the lowest, so it stays BACKUP while the
PVE nodes exist and wins the VIP by default once they are gone, with no
config change at cutover.
Verified: playbook syntax-checks; ntp_cluster resolves to the four hosts;
Arch paths confirmed against the package file list. Not yet run against any
host, and runtime var resolution is unverified (needs the vault password).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
external-dns was the last Cloudflare consumer without rotation automation. It
now holds a dedicated single-zone token, so it can roll like the cert-manager
one: PUT /user/tokens/{id}/value keeps scope and policy, changes the value.
Unlike the cert-manager rotator, this needs Reloader — external-dns reads
CF_API_TOKEN as a startup env var. The annotation added in #311 is exercised
by this job and confirmed working: Reloader restarted the Deployment 10s after
the Secret changed, with no manual intervention.
Reuses the existing cloudflare bootstrap so the admin-token lives in one place,
but pins a SEPARATE token-id (external-dns-token-id) so the two tokens roll
independently. Verified: rolling external-dns left the cert-manager token
untouched and all 27 Certificates True.
ExternalSecret refresh 1h -> 10m to match the rest of the rotation fleet.
Tested end-to-end: job completed in 5s, OpenBao patched, ESO synced, Reloader
restarted the pod automatically, external-dns resumed with 0 errors, and the
stored value verifies as the correct active token with no trailing newline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live inspection of the PVE cluster on 2026-07-19 found two defects.
1. No IPv6 address on any external interface, on all three nodes.
/etc/frr/frr.conf enables "ipv6 forwarding" for the OSPFv3 Thunderbolt
mesh that carries corosync, which sets net.ipv6.conf.all.forwarding=1.
The kernel default accept_ra=1 means "accept RAs only while NOT
forwarding", so the nodes silently discarded the Router Advertisements
the gateway sends on VLAN 40. Every external interface was left with
only a link-local address; the sole non-link-local IPv6 anywhere on the
nodes was the corosync mesh /128, which is static in frr.conf and never
depended on RAs.
Verified the gateway is not at fault: RAs arrive on vmbr1.40 from
fe80::265a:4cff:fea6:fd3c every ~10-15s, carrying both the Telekom GUA
prefix and a ULA, and non-forwarding hosts on the LAN autoconfigure
normally. The PVE nodes are the only hosts running FRR.
Fixed with a sysctl drop-in setting accept_ra=2, the value a forwarding
host needs in order to still autoconfigure its own uplink.
2. saturn's vmbr0 is at MTU 1500 while its own uplink enp2s0f0np0 and both
peer nodes' bridges are at 9000. Since the three bridges share one L2
domain, large frames between saturn's guests and the other nodes' guests
blackhole rather than fragmenting. saturn's vmbr0 stanza is simply
missing the mtu line that jupiter and neptune both have.
The two fixes are separated by risk. The sysctl change is non-disruptive
and applies on every run. The MTU change is staged as a pending PVE network
change and applied only when pve_node_network_apply_pending is set, because
applying it reloads networking on the node and briefly interrupts the Talos
k8s VM and its Ceph OSD. The play uses serial: 1 so nodes converge one at a
time.
The MTU task reconstructs the full bridge definition from a pvesh GET before
issuing the PUT. A PUT on /nodes/<node>/network/<iface> replaces rather than
merges, so passing only --mtu would silently drop bridge_ports,
bridge_vlan_aware and bridge_vids, detaching the uplink and flattening every
SDN VNet.
Not applied yet — no run has been made against the cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous comment asserted the annotation "must" sit on the Deployment
rather than the pod template. That is wrong. An A/B test (two identical
Deployments consuming one Secret, differing only in annotation placement)
showed Reloader acting on BOTH within 10s, naming each in its log.
deploymentAnnotations is still used here, but only to match the prevailing
idiom in this repo — not because pod-template annotations are ineffective.
This also clears the suspicion raised against authentik-server,
authentik-worker and crowdsec-lapi: their placement is fine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
external-dns was moved off the shared 4-zone "Caddy" Cloudflare token onto a
dedicated single-zone token (Kubernetes External-DNS: Zone Read + DNS Write on
renner.uno only). Rotating that Secret alone is not enough: external-dns reads
CF_API_TOKEN as an env var at startup, so it kept serving with the old token
until the pod happened to restart. During this cutover that restart had to be
done by hand.
Add the Reloader annotation via deploymentAnnotations so a future rotation
restarts the Deployment automatically.
Placement matters: the annotation goes on the Deployment, not the pod template.
Reloader has only ever been observed acting on workload-level annotations in
this cluster (woodpecker, paperless, netbird-ingress, pve-exporter); the chart
exposes deploymentAnnotations for exactly this.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#306 landed suspended pending a privsep check and a bootstrap credential.
Both are now done and the job is verified, so enable the schedule.
privsep caveat RESOLVED: pve-exporter@pve!monitoring is privsep=0 and its
ACL (/ -> PVEAuditor) is on the USER, not the token. DELETE+re-create with
privsep=0 therefore loses no permission and needs no ACL re-grant.
Bootstrap credential is a dedicated least-privilege PVE identity rather than
root@pam. PVE rejects per-user ACL paths and authorises the token endpoints
via a userid-group check, so the tightest available scope is a group whose
only member is the target user:
role SecretRotator (User.Modify)
user secret-rotator@pve
ACL /access/groups/secret-rotated -> SecretRotator
Verified it can mint/delete pve-exporter tokens, sees no VMs or storage,
and gets 403 minting a root@pam token.
api-url is the in-cluster Service (all three nodes back it). The external
name pve.renner.uno cannot be used from a pod: pods cannot hairpin to their
own cluster's ingress LB, which hangs the job. A node IP or single node
FQDN would be a needless SPOF.
Tested: two full rotations; OpenBao patched, ESO synced, Reloader restarted
the exporter, pve_up returned 1.0 for all three nodes both times.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Staged verification found cert-manager was using an unrelated 4-zone token named 'Caddy' (renner.uno + 3 personal zones, shared with something outside this repo), not the dedicated 'Kubernetes Cert-Manager' token that already existed (renner.uno only). Switched cert-manager to the dedicated token and pinned the rotation token-id to it, so Caddy is never rolled. Net: cert-manager drops write access to 3 personal zones.
Verified end-to-end: job rolls token af708481, OpenBao rotates, rolled token stays active/same scope, ESO syncs, cert-manager writes the DNS-01 TXT record (publicly resolvable), 27/27 prod certs Ready. suspend: false.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Same fix proven on netbird (#307/#308): alpine+apk -> python:3.13-alpine stdlib so it runs under the non-root restricted PSA. Uses a dedicated no-verify SSL context for the self-signed Proxmox API only (OpenBao stays plain http). Still suspended + untested; the privsep=0 caveat + brief monitoring gap on DELETE+recreate remain documented in the header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Applies the same fix proven on netbird (#307/#308): switch alpine+apk to python:3.13-alpine stdlib so the job runs under the non-root restricted PSA. CF's roll is in-place (PUT /tokens/{id}/value) so no key-settings pitfall like netbird had. Still suspended + untested (needs the bootstrap admin-token + token-id).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Testing #304 end-to-end revealed the minted setup key had the WRONG settings vs the ingress key: ephemeral=false (should be true — peers must be auto-reaped; these DaemonSet peers re-enroll fresh each restart) and allow_extra_dns_labels=false (should be true — the DNS capability). Remediated live and fixed the payload:
- add ephemeral: true, allow_extra_dns_labels: true to the setup-keys POST.
- NB_AUTO_GROUPS = d8umic2oc9cs73d4e1f0 (k8s-ingress-router).
- KEEP the 1y expiry (intentional: an auto-rotated key should expire so a broken rotation surfaces — do NOT match the original's never-expire).
- suspend: false (tested + enabled).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Testing #304 revealed the runtime 'apk add curl jq' fails ('Unable to lock database: Permission denied') because the job runs non-root (runAsUser 65534) and the openbao ns enforces restricted PSA — apk needs root. Switch to python:3.13-alpine and do the HTTP/JSON via stdlib (urllib/json), no package install, non-root-safe. Same fix must be applied to the still-open cloudflare (#305) and proxmox (#306) Class-B jobs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Rotates pve-exporter@pve!monitoring via the Proxmox API. Proxmox has no in-place secret roll, so DELETE+re-create (privsep=0), regenerate pve.yml with the new token_value, patch OpenBao. openbao-ns pattern; per-secret rotator-proxmox role reads a bootstrap privileged Proxmox token + patches pve-exporter-config. refreshInterval 1h->10m; Reloader restarts pve-exporter.
SHIPPED SUSPENDED + UNTESTED (riskiest Class-B): needs a bootstrap privileged Proxmox token at homelab/infrastructure/secret-rotators-bootstrap/proxmox, VERIFICATION that the token uses privsep=0 (else re-grant its ACL after re-create), and awareness of the brief monitoring gap during DELETE+create. Test before spec.suspend=false.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Rolls the cert-manager DNS-01 Cloudflare token in place (PUT /user/tokens/{id}/value) then patches OpenBao. openbao-ns pattern; per-secret rotator-cloudflare role reads a bootstrap admin-token+token-id and patches the api-token path. refreshInterval 1h->10m; no Reloader (cert-manager reads fresh per challenge).
SHIPPED SUSPENDED + UNTESTED: needs the bootstrap admin-token (Edit User API Tokens perm) + the cert-manager token-id provisioned to homelab/infrastructure/secret-rotators-bootstrap/cloudflare, then a manual test (issue/renew a cert) before spec.suspend=false. Risk: a bad roll breaks TLS issuance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
First Class-B rotation: calls the self-hosted NetBird mgmt API to mint a fresh reusable setup key, then patches OpenBao (ESO syncs, Reloader restarts the netbird-ingress DaemonSet to re-enroll). openbao-ns pattern; per-secret rotator-netbird role/policy that reads a bootstrap NetBird PAT + patches the setup-key path. refreshInterval 1h->10m.
SHIPPED SUSPENDED + UNTESTED: needs (1) the bootstrap PAT provisioned to homelab/infrastructure/secret-rotators-bootstrap/netbird, (2) NB_AUTO_GROUPS filled with real NetBird group IDs, (3) a manual on-demand test, before spec.suspend is set false. Class-B template for the cloudflare + proxmox rotations to follow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Rotates crowdsec-bouncer-key:key quarterly + on-demand via the openbao-ns pattern (per-secret rotator-crowdsec-bouncer role/policy, no netpol). Self-sync: the single KV path feeds both the crowdsec + ingress ns Secrets; Reloader restarts crowdsec-lapi. Both ExternalSecrets refreshInterval 1h->10m.
CAVEAT handled: traefik/crowdsec-middleware.yaml has a THIRD hardcoded copy of this key (not Secret-sourced) — that middleware is inert, and a loud guard comment now warns it must be refreshed from OpenBao before re-enabling. Requires the one-time OpenBao bootstrap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Rotates authentik-secrets:secret-key (Django signing key) quarterly + on-demand via the openbao-ns pattern (per-secret rotator-authentik role/policy, no netpol). Self-sync 2-sided: server + worker read the same secret; Reloader restarts both via global.podAnnotations. smtp-password in the same KV entry is preserved. externalsecret refreshInterval 1h->10m. BLAST RADIUS: invalidates active SSO sessions + in-flight signed links (re-login); downstream LDAP/proxy outposts unaffected. Requires the one-time OpenBao bootstrap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Moves the WOODPECKER_AGENT_SECRET rotation from the apps namespace to the openbao namespace (matching the new paperless prototype), so it reaches OpenBao intra-namespace:
- openbao/rotation/woodpecker.yaml: SA + CronJob now in openbao ns, per-secret role rotator-woodpecker.
- openbao/policies/rotator-woodpecker.hcl: replaces the shared rotator policy (retired); per-secret least-privilege.
- REVERTS the #299 netpol pair (apps egress + openbao ingress) — no longer needed intra-ns.
- deletes apps/woodpecker/rotate-agent-secret.yaml and the old rotator.hcl.
server.yaml/agent.yaml (Reloader annotations) and externalsecret.yaml (10m) are unchanged.
Bootstrap: create rotator-woodpecker role/policy in openbao and delete the old rotator role/policy (documented in rotator-woodpecker.hcl).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
First fan-out of the secret-rotation pattern proven on woodpecker, now using the cleaner openbao-namespace design: the rotation Job runs in the openbao ns so it reaches OpenBao intra-namespace — NO NetworkPolicy edits needed (unlike woodpecker's apps-ns job). Per-secret isolation: its own scoped policy+role (rotator-paperless) so a compromised job can only touch the paperless KV path.
- openbao/rotation/paperless.yaml: SA + quarterly/on-demand CronJob (openbao ns) that bao kv patch's secret-key at homelab/apps/paperless (DB creds preserved).
- openbao/policies/rotator-paperless.hcl: least-privilege policy (manual bootstrap, documented in-file).
- paperless.yaml: reloader annotation; externalsecret.yaml: refreshInterval 1h->10m.
Rotating the Django SECRET_KEY invalidates paperless sessions (re-login); no data loss. Requires the one-time OpenBao bootstrap before first run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The woodpecker-secret-rotate CronJob (#297) runs in ns apps and could not reach OpenBao (i/o timeout on auth/kubernetes/login) — blocked on BOTH sides: apps is egress-default-deny (no openbao allow) and openbao is ingress-default-deny (rotator not allowlisted). The snapshot job avoids this by living in the openbao ns; ESO is explicitly allowlisted.
- default-deny.yaml (apps egress): namespace-wide allow to openbao:8200, scoped in effect by openbao's ingress (mosquitto pattern).
- openbao-ns.yaml (openbao ingress): scoped allow from the rotator pod labels (apps/woodpecker/secret-rotator) — the real security boundary.
Verified apps has no other blocking egress; kustomize build + yamllint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Closes the H2 gap from the rotation PR (#297) adversarial review: the rotation Job is fire-and-forget, so a downstream failure (ESO/Reloader/pod-reschedule) yields a green Job with no signal. ESO metrics aren't scraped here, so these are kube-state-metrics based:
- WoodpeckerServerNotReady (critical): server StatefulSet 0 ready for 20m.
- WoodpeckerAgentNotReady (warning): agent Deployment 0 available for 20m.
- WoodpeckerSecretRotationJobFailed (warning): named signal for a failed rotation Job (e.g. missing OpenBao bootstrap).
- WoodpeckerSecretRotationStale (warning): quarterly CronJob hasn't succeeded in >100d; absent-until-first-success so it can't false-fire early.
Independent of #297: readiness alerts are useful immediately; the Job/CronJob series stay empty until that PR merges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
- H1: pin Reloader chart >=1.0.0 <2.0.0 (was open upper bound; a cluster-wide restart-authority controller must not float across majors — repo pinning discipline).
- M2: correct the restart-downtime comment (server StatefulSet is unavailable for tens of seconds, not 'within seconds').
- M1: note the on-demand/emergency rotation runs during incidents (the 'outside CI hours' safety doesn't apply there).
- H2: document the no-convergence-check limitation + follow-up (add rollout/health check or alert before scaling the pattern).
- L1: correct rotator.hcl rationale — native 'bao kv patch' is a server-side merge, needs no client read-modify; 'read' is surplus.
- L2: warn against token_no_default_policy=true (breaks the kv-v2 preflight).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
End-to-end cluster-internal rotation of the shared server<->agent gRPC secret, no per-rotation human step:
- rotate-agent-secret.yaml: quarterly CronJob (+ on-demand) logs into OpenBao via k8s-auth role 'rotator', generates a random value, bao kv patch's homelab/apps/woodpecker (Gitea OAuth fields untouched).
- infrastructure/reloader: Stakater Reloader restarts annotated workloads on Secret change.
- server.yaml/agent.yaml: reloader annotation for woodpecker-secret.
- externalsecret.yaml: refreshInterval 1h -> 10m so a rotation propagates within ~10m.
- openbao/policies/rotator.hcl: least-privilege policy (patch on the one path only); NOT Flux-applied, one-time manual bootstrap documented in-file (eso-read.hcl convention).
PoC scope: woodpecker only. Requires a one-time OpenBao bootstrap (rotator policy + role) before the CronJob succeeds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
No hardcoded IP remains in the pve scrape config: the node job now
targets jupiter/neptune/saturn.renner.uno. instance label reads the
hostname, and a renumbered node follows DNS without a config edit.
Verified: exporter returns node-local metrics when targeted at
jupiter.renner.uno from inside the cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The pve cluster-collector job hardcoded 10.2.40.201, so cluster-wide
metrics (pve_up, guest info, storage) paused whenever jupiter
specifically was down, even with the cluster healthy. Point it at the
pve Service DNS (pve.apps.svc.cluster.local) whose EndpointSlice
load-balances the PVE API across all three nodes, so the scrape
follows whichever node is up. Node-local job keeps the three explicit
targets — it needs per-node identity.
Verified: exporter returns pve_up for cluster + all nodes when
targeted at the Service DNS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The single pve scrape job hit all three node IPs with cluster=1, and
pve-exporter returns the full cluster-wide resource set regardless of
which node answers - every node/guest series landed in the TSDB three
times (same id, differing instance). Dashboards papered over it with
max by (id); the cardinality cost stayed.
Split per upstream prometheus-pve-exporter guidance: cluster collectors
scrape one node only, node-local collectors keep scraping all three
under a new pve-node job. No dashboard queries filter on job=, and all
already aggregate by id, so this is query-compatible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every panel was dead: all 11 datasource refs pointed at the
prometheus-legacy datasource (retired 2026-05-09). Repoint to the
active prometheus datasource.
The alloy job also carries no 'host' label, so all legends and the
Agent Versions table rendered empty labels. Join via kube_pod_info
(pod -> node) so DaemonSet panels are labelled k8s-1/2/3, collapse
the eval-time histogram per pod before quantile, and tidy the
versions table to node/pod/version columns.
All queries verified against live Prometheus (3 agents, one clean
series per node).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All 10 panels were dead: the dashboard queried datasource uid
prometheus-legacy (prometheus.srv.renner.uno:9090), which no longer
resolves (Grafana proxy: 502 dial tcp lookup i/o timeout). Gitea now
runs in-cluster and is scraped by the in-cluster Prometheus
(job="gitea", instance gitea.apps.svc.cluster.local:3000), so switch
every panel to uid prometheus.
Also wrap all targets in max(): the pre-migration instance
gitea.srv.renner.uno:3000 stopped 2026-07-12, and windows spanning
that date would otherwise render two identically-named series per
query. Stat panels additionally get instant:true so a dead exporter
shows No data instead of a stale last value.
- Historical Cost Comparison: the chained joins renamed _value columns to
_value_j2/_value (not _value_gi/_value_sp as the map expected), so the
solar ratio was always null->0 and the whole cost line was priced at full
grid price (EUR 4.67K instead of ~3.4K). Streams now pre-rename columns
to h/d/gi/ge/sp so the join is unambiguous.
- Blended price now uses counter deltas since heat-pump install (Oct 2024)
instead of device-lifetime counters that include 1.4 years of PV-only
history from May 2023 (ratio 0.369 vs 0.322 -> EUR 0.236/kWh vs 0.248).
Applied consistently to all cost/savings/ROI stats and the cost chart
(chart subtracts install-date counter baselines).
- Annual savings / payback: replace the calendar-YTD extrapolation
(12 / (month + day/30) - which overcounts elapsed time by a month and is
seasonally biased) with the actual trailing-365-day consumption spread.
- Fix stale legend 'Last' values (10734/1230/EUR 9.47K): the counters'
friendly_name tag changed 2026-03-04, splitting each series into two
tables; group()+sort() merges them before aggregateWindow.
- New oil_baseline dashboard variable (6650) replaces six hardcoded copies.
- Running Cost: neutral color (was green/yellow/red on a forever-growing
cumulative), retitled '(Since Install)'; added descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- App Servers Up: job="authentik" matches nothing (real job label is
authentik-server-metrics) -> panel showed 'No data'. Also 'or vector(0)'
+ red threshold so a dead scrape shows a red 0 instead of No data.
- Task/Admin Workers, Flows/Policies Cached: raw gauges return 4 series
(2 pods x 2 pids) -> stats rendered 8 repeated numbers. Wrapped in max().
- Outposts Connected: raw per-outpost/per-pid series rendered an unreadable
grid of ~30 0/1 values (0s were per-process views, not real disconnects).
Now counts outposts with >=1 active WebSocket connection (10/10).
- Outpost Connections: legendFormat used non-existent labels {{name}}/{{uid}}
(real: outpost_name/uuid) -> every legend row was '-'. Replaced with
server-side authentik_outposts_connected as a state-timeline, one row per
outpost, red mapping for 'getrennt'; covers all 10 outposts instead of the
4 with scraped sidecar metrics.
- All stat queries switched to instant so dead sensors render honestly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Nodes Online/Down used job="node" but the scrape job is "node-exporter":
both stats showed No data. Nodes Down additionally gets OR on() vector(0)
so it renders 0 instead of No data when all nodes are up, and green-at-0/
red-at-1 thresholds (was permanently red).
- Avg Disk Usage + Disk Usage by Node filtered mountpoint="/" which does not
exist on Talos (only /var is a real xfs filesystem): both panels were empty.
- Disk I/O summed all block devices including the ~12 Rook-Ceph rbd* mappings
per node (~438 kB/s vs 8-129 kB/s physical): filter to nvme.*|vd.*.
- Network I/O filter was written for a Docker host (veth/docker/br-) and let
~40 Cilium lxc* per-pod interfaces plus kernel pseudo-devices through,
double-counting pod traffic: filter to eth.* (bond0 is down, 0 slaves).
- Legends showed instance IP:port; join node_uname_info via
on(instance) group_left(nodename) so panels label k8s-1/2/3. Both join
sides are wrapped in max by (...) so historical duplicate series from
pod restarts / the pre-2026-07-16 duplicate-scrape residue cannot break
the join or rainbow-duplicate long-window views.
- Stat panels reduce lastNotNull instead of mean: show current value, not
a time-average over the selected window.
All queries verified against live Prometheus before applying.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dashboard was 100% dead: every panel still referenced the
prometheus-legacy datasource retired on 2026-05-09, so all panels
rendered 'No data' with datasource errors.
- Repoint all panels prometheus-legacy -> prometheus
- Remove TCP Endpoint Status panel: the blackbox-tcp Probe was
deliberately deleted 2026-05-20 (see probes-blackbox.yaml), the job
no longer exists
- Remove DNS Status panel: no blackbox-dns job has ever existed in
this Prometheus (the dns-servers job is Alloy on the DNS VMs, not a
probe)
- Replace the freed row with HTTP Status Codes (probe_http_status_code,
instant, 2xx green / 3xx orange / 4xx+ and probe-fail red)
- Convert SSL Certificate Expiry from a seconds-countdown timeseries to
a days-remaining bar gauge (instant, red <7d, orange <30d, green)
- Make status stat queries instant so dead probes render No data
instead of a stale last-known value
- Pin Uptime y-axis to 0-100% and drop the fill (10 overlapping series
at 100% rendered as a muddy block)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pve-exporter is scraped via all 3 PVE nodes and each scrape returns the
full cluster view, so every series exists in triplicate. All stat counts were
3x too high (Online Nodes showed 39 instead of 3) and every bargauge/table
row appeared three times. Wrap all queries in max by (id) / count by (id).
- Online Nodes: count unique node/* ids with pve_up==1 (was 39, now 3);
threshold green at 3 so a lost node turns red
- Total VMs: count unique qemu guests excluding templates (was 12, now 3)
- Total LXC: count unique ids (was 9, now 3)
- Running Guests: pve_guest_info has no status label in this exporter
version (panel showed No data); derive from pve_up==1 per guest instead
- Node CPU/Memory bargauges + timeseries: filter to id=~"node/.*" — they
showed every guest (mostly lxc rows), not nodes
- Guest Status table: dedupe rows, add Status column from pve_up (color
mapped 1=Running/0=Stopped), drop useless Info column, migrate
custom.displayMode -> custom.cellOptions (schemaVersion 38 skips the
Grafana 10 migration, so gradient gauges/color background were silently
ignored)
- Node Disk I/O -> Guest Disk I/O: pve_disk_read/write metrics only exist
for qemu/lxc guests, never node/* (panel was permanently No data); use
the *_total counters with rate()
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dashboard was entirely dead: every panel queried postgres_exporter
metrics (pg_up, pg_stat_activity_count, ...) that have never existed in
this Prometheus. Postgres runs as four CloudNativePG clusters (ak-pg,
im-pg, gf-pg, pprl-pg) whose built-in exporter on :9187 was not scraped
at all (enablePodMonitor is false and no PodMonitor existed).
- add PodMonitors for the four CNPG clusters in monitoring-rules,
labelled release=kube-prometheus-stack so the Prometheus
podMonitorSelector actually picks them up (CNPG's own
enablePodMonitor would generate unlabelled PodMonitors that this
Prometheus ignores)
- rewrite all dashboard queries to the cnpg_* metrics the exporter
actually exposes (verified against the pods' /metrics endpoints)
- dedupe multi-instance clusters (max by namespace/datname for sizes,
primary-only filter via cnpg_pg_replication_in_recovery == 0 for
per-database counters)
- replace the Active Locks stat (no lock metric in the CNPG default
queries) with Max Replication Lag
- instant queries on all stat panels so dead instances show No data
instead of a stale last value
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Healthy Hosts: job 'integrations/unix' does not exist here -> node-exporter;
thresholds now red/yellow/green(3) so a single down node shows yellow
- Unhealthy Services: 'count(up == 0)' returns empty vector when healthy ->
'or vector(0)' renders an honest 0; both stats now instant queries
- CPU: 'avg by(host)' collapsed all 3 nodes into one series (no host label) ->
avg by(instance) joined with node_uname_info for k8s-1/2/3 legend names
- Memory: legend showed raw label dumps ({{host}} does not exist) -> nodename join
- Disk: Talos has no '/' mountpoint in node_filesystem -> use /var (the
persistent data disk); retitle panel accordingly
- Cumulative Value: Self-Use Savings and Total Value series rendered nothing.
The union'd solar/export/price tables had mismatched group keys (pivot ran
per-table) AND the sparse strompreis series almost never landed in the same
aggregation window. Fix: group() before pivot + fill(usePrevious) per column.
Verified via /api/ds/query: 471 points, final value EUR 4363 == stat panel.
- Solar Power stat: was aggregateWindow(mean) over the dashboard range (~17 h
windows at 3 y), showing 3.04 kW at midnight. Now last() -> honest 0 W at night.
- Solar Installed: HA input_number stores 8140 (Wp, mislabeled kWp); with unit
kwatt the panel would read 8.14 MW. Divide by 1000 -> 8.14 kW, documented.
- Timeseries gap honesty: e3_* series have a real data hole (Nov 2025 - Feb 2026,
verified 0 points/month). createEmpty:false let Grafana draw fake decay ramps
across it. Now createEmpty:true so gaps render as gaps (panels 20/21/22).
- Units: 'kwh' is not a valid Grafana unit id (rendered lowercase 'kwh');
replaced with 'kwatth' everywhere -> proper kWh/MWh scaling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Current by Phase: regex /smart_meter.*current/ matched nothing (smart
meter exposes no current entities) -> use verified
e380_energy_meter_e380_current_l[123]; spanNulls for sparse updates
- Energy Consumption: measurement "kWh" does not exist in the bucket
-> plot verified Wh daily counters (grid import/export, PV production,
battery charge/discharge) with readable display names
- Active Lights: light-domain data is never recorded under measurement
"%" -> replace dead table with Lights/Standby/Switches power
timeseries from verified powercalc aggregates (all_*_power)
- Top 20 Devices: add group() so sort/limit work globally (was
alphabetical, unsorted, per-table); exclude grid meters, PV/battery,
forecasts, powercalc room/floor aggregates and _device_power twins;
fresh 30-min window instead of stale last() over the full range;
rowsToFields for clean labels (was "_value xyz_power")
- Grid semantics: retitle Total Power -> Grid Power (+Import/-Export),
blue threshold for export; unstack mixed-sign phase graph; short
L1/L2/L3 legend names; taller voltage/current panels so the legend
table is not cut off
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Alarm stats: '== 1' dropped the series when false, so Armed Home/Away and
Triggered could only ever show 'No data' and never their 0-state; Disarmed
rendered 5 overlapping series. Use min()/max() aggregation over the one-hot
alarm_control_panel_state series so every tile always shows a real 0/1.
- Windows/Doors tables: replace friendly-name regexes (matched battery-low
sensors, HmIP 'Window/door is CLOSED' entities with inverted semantics,
intrusion/tamper diagnostics, _2 twins, pond-pump/fridge sensors) with a
curated entity list: one canonical is-open entity per physical sensor.
Collapse scrape labels via max by so the tables show Sensor/Name/Status.
- Motion panels: old regex required 'Bewegungs.*Occupancy' and missed the
Eingang sensor (typo 'Bewegunsmelder'), all motion-class sensors and the
mmWave Praesenzmelder. Curated list of physical detectors.
- Covers: name-based exclusion (.*Nord.*|.*Ost.* etc.) dropped 8 physical
covers incl. the garage door while letting the eg_schlafzimmer/eg_kuche
group entities through. Exclude the 5 group covers by entity instead;
label bars by entity because friendly names repeat ('Rollo' x3).
- Batteries: filter battery_notes '+' twin entities, drop PV storage SoC,
remove '> 0' (which would hide a 0% dead battery), sort ascending so the
most critical batteries are on top, instant query so stale values drop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The template variable queried label_values(node_uname_info, host), but no
node-exporter metric in this Prometheus carries a 'host' label (that label
only exists on blackbox-probe metrics). The variable resolved to empty and
all five panels rendered blank.
- Replace $host with a visible $node variable (nodename: k8s-1/2/3) plus a
hidden chained $instance variable (nodename -> instance via node_uname_info)
- Filter Cilium pod interfaces (lxc*, cilium_*) and always-zero kernel
pseudo-devices (ip6tnl0, sit0, teql0, tunl0) out of Network I/O
- Filter Ceph RBD-mapped volumes (rbd0..11) out of Disk I/O so the panel
shows physical disks (nvme0n1, vda) only
- Exclude the rootfs bind mount (/etc/extensions.yaml) from Filesystem
Usage, leaving the meaningful /var xfs mount
All queries verified against live Prometheus before commit.
- All temperature/humidity panels: replace vendor regexes (shelly.*temp etc.)
that dragged in device-internal temperatures (pond pump 34C, power meter 60C,
relay 76C) with explicit allow-lists of verified room sensors
- Add missing sensors: shelly_blu_h_t_7203 (KG Schlafzimmer),
treppenhaus_eg_bewegungsmelder (EG Treppenhaus), vento humidity sensors,
Shelly Wall Display (DG Buero)
- OG & DG panel: regex had no og_ alternative and showed only device internals;
now honest DG-only series + description (OG deliberately has no sensor)
- Heating System / Hot Water panels were empty (domain climate/water_heater
does not exist in the C measurement) -> use vitocal_250_a_* sensors
- Current-value panels: fixed -6h window so dead sensors (EG Shelly H&T trio,
DG Meter Pro) disappear instead of showing stale values as live
- Legends: map entity_ids to room names in Flux + displayName, removes the
'_value xyz' prefix
- Align CO2 thresholds between gauge and history (800/1000/1500)
- Server node: sensor.keller_server_power is dead (UPS retired) -> Wave PM
Mini server outlet (kg_vorratskeller_wave_pm_mini_server_steckdose)
- Garden node: sensor.eg_garten_teichpumpe_power_2 is dead ->
sensor.shelly_teichpumpe_power_0 (live, peaks ~77 W)
- Home node/series: floor sums only cover submetered circuits (702 W shown
vs ~1.1 kW real); Home is now the meter balance PV + Grid + Battery, with
a new grey 'Rest' node for the non-submetered remainder
- Flow lines: Heat Pump is metered inside Keller (was drawn from EG);
Garden is on floor Aussenbereich, in no floor sum (was drawn from OG)
- Drop redundant batteryIn target (derived from the signed battery sensor);
update stale comments from the retired *_absolute sensors
- History: max by (entity) guards against duplicate-scrape residue that
doubled the summed Home series; sign hints in Battery/Grid legends;
grid query pinned to entity id instead of friendly_name
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Status panel: job label is home-assistant (hyphen), was homeassistant -> No data
- Doors & Windows / Motion: exporter has no device_class label; select curated
entity lists instead (one canonical is-open entity per physical contact,
physical motion/presence sensors without zone/magic_areas duplicates)
- Temperature: was all 61 temp series incl. device-internal electronics temps
(fridge 5C, PoE switch 75C, Shelly 58-75C); curated to room/ambient sensors
+ outdoor temp, incl. currently-dead EG H&T sensors so they reappear
- Power Consumption: exclude non-consumption series (Solcast forecasts,
battery max charge/discharge capability, derived night-load stats)
- Energy: cumulative lifetime counters (up to 80 MWh) told nothing; now
bargauge of top 12 increase() over 24h, forecast counters excluded,
max by () dedup against scrape residue
- Switches: 248 unreadable micro-tiles -> single 'Switches On' count stat
- Battery: bottomk(12) instant, battery_notes *_battery_plus twins excluded
- Entity counts: max by (entity) dedup, instant queries on all stats
- Unavailable Entities: neutral blue; ~1139 is the structural baseline
(verified against HA API), old yellow>=1/red>=5 thresholds were meaningless
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All queries selected raw series by entity label only; over a 90d window HA pod
restarts plus the (since fixed) duplicate proxy scrape produce 30 distinct
series per entity, rendering the Außentemperatur & Puffer panel as an
unreadable 30-entry legend jumble. Wrap every query in max() so exactly one
series per entity survives — this also keeps the xychart joinByField transform
correct once heating data arrives, since the join needs unique field names.
The stat panel now says 'keine Heizdaten im Zeitraum' instead of a giant
'No data': the logging sensor is intentionally unavailable outside active
heating loading, so an empty summer dashboard is by design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All four panels queried raw HA metrics that still carry ~12 duplicate
series per entity in the 90d window (two scrape jobs x six pod
lifetimes; the duplicate-scrape bug was fixed 2026-07-16 but the
residue stays in range queries until it ages out). This broke the
scatter join (no points rendered), made the stat panel show 8
conflicting averages, and drew rainbow duplicates in both time series.
Wrapping every target in max by (entity) (...) collapses the residue
to one series per entity; values between the duplicate jobs were
near-identical, so max is lossless.
- Windows/Doors/Batteries tiles: literal ${__value.raw} never interpolates in
value mappings; replaced range mappings with suffix units. Counts now use
curated entity lists (one canonical 'is open' entity per physical sensor)
instead of friendly-name regexes that matched inverted *_is_closed sensors,
'Battery low' sensors and duplicate _2 entities.
- Alarm tile: old min(...==1) query could never show TRIGGERED; now
max(...{state="triggered"}) with 0=OK/1=TRIGGERED mapping.
- Power Usage: old regex summed phases A+B+C AND total_active_power AND
device_power -> ~2x real value; now reads total_active_power only.
- Floor temperature panels: select verified room sensors by entity id instead
of vendor regexes (which pulled 65-75C device-internal temps and missed the
I-9PSL sensors that live in the EG). Instant queries so dead sensors do not
render stale values. OG: no room sensor exists (owner-confirmed) -> honest
noValue; DG SwitchBot offline -> noValue 'Sensor offline'.
- Tables: dedup via max by (entity, friendly_name), drop k8s scrape label
columns, colour only the value column; battery list excludes battery_notes
*_battery_plus twin entities and no longer hides 0% batteries.
- Power Consumption: L1/L2/L3 legends, Total from total_active_power.
- Indoor Temperatures: room sensors only (y-axis no longer squashed to 80C).
- Heating row: Hot Water/Heating pointed at post-open3e-migration vitocal
sensors (old queries returned nothing resp. only the fridge); humidity/CO2
deduplicated; row renamed to Heating & Air Quality.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The union-based timeseries series were named via set(_field, ...), which Grafana's InfluxDB frontend rendered as "_value {_start=...}" instead of the intended labels (and broke the byName color overrides). Data was correct; only the legend/colors were affected.
Switch to the keep(columns:[_time,_value]) |> rename(_value -> "Name") pattern (as used in heatpump-roi), which names each series cleanly. Verified in Grafana Explore. No data/logic change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
The 2026-07 ViCare->open3e/MQTT migration renamed the Vitocharge PV/battery entities (e3_vitocharge_03_* -> vitocharge_vx3_vx3_*) and changed some units. Both dashboards still queried the retired entities and showed No data.
power-flow (Prometheus, real-time): solar_power->pv_power, battery SoC->battery_soc; the two absolute battery-power sensors are replaced by a single signed battery_power (open3e: positive=discharge, negative=charge), so charge/discharge derive from its sign.
pv-system-roi (InfluxDB): repoint every query; fix units (PV/battery energy now Wh not kWh -> /1000; grid energy now kWh not Wh); grid power sign unchanged (negative=export). Timeseries panels UNION pre-migration e3_* with post-migration series (unit-normalized) so history back to 2023 stays unbroken. Reserved Energy recomputed as capacity x backup_discharge_limit. strompreis + the two input_number config entities survived and are kept.
Validated live via Grafana Explore: signed-battery charge/discharge union, unit normalization, pivot/value patterns (reused from merged heatpump-roi fix#271).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
Both destroy applies (runs 1180/1181) died instantly with 'context
deadline exceeded' on the pre-destroy status GET while the PVE API
answers in <1s: the bpg provider reads timeout_delete from STATE at
destroy time, and the stage-1 import stored 0 for the timeout-less
shims — a zero-second context. Read paths carry their own timeout,
which is why import and plan were clean.
v2 declares all timeouts explicitly (one in-place, no-API update
writes them into state) and mirrors the real container attributes
(unprivileged=true is ForceNew — mirroring prevents a replace).
No ignore_changes this time: the timeout update must actually apply.
Flow: merge -> plan (must be in-place only) -> apply -> follow-up PR
deletes the file -> destroy apply.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The ViCare cloud integration was replaced by open3e/MQTT (2026-07), renaming all heat-pump/PV entities (e3_vitocal_*/e3_vitocharge_03_* -> vitocal_250_a_vitocal_*/vitocharge_vx3_vx3_*). The dashboard still queried the retired IDs, and a duplicate Prometheus scrape broke the surviving-entity panels with many-to-many errors.
Rebuild every panel on InfluxDB (bucket suedstr28, authoritative + live) using current entity IDs. Where an entity was renamed, UNION the pre-migration e3_* series with the post-migration series (unit-normalized: PV Wh->kWh) so history stays unbroken. Drop all Prometheus targets. Add a grid_price variable (0.3271) replacing the retired live strompreis read.
Also rename the home-assistant-bgp Service port http->ha so the home-assistant ServiceMonitor no longer scrapes HA a second time via the LB (:8123), which duplicated homeassistant_sensor_* series cluster-wide.
Queries validated live via Grafana Explore: blended price 0.248 EUR/kWh, total consumption 14284 kWh, projected annual savings 4698 EUR, oil proration 11912 EUR, old-union-new grid-import continuous.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
Stage 1 (#268) adopted the never-tofu-managed rustfs containers into
state (import run 1173: all three imported, plan 'No changes').
Removing the shims makes the next apply destroy the containers —
guest lifecycle fully through tofu, per owner rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The live 15-min snapshot structurally misses anything that fires and
resolves between ticks: on 2026-07-16 a 17-minute KubeHpaMaxedOut
(authentik) and last evening's four-alert home-assistant incident were
invisible to the monitor (owner noticed the blips; monitor did not).
The fetch step now also queries Prometheus's ALERTS history over a
35-minute window (~2 ticks + jitter). The dedup step synthesizes
pseudo-alerts for series not currently firing at all — checked against
the PRE-filter snapshot so live-but-null-routed alerts don't reappear
as blips — with a stable sha256-derived fingerprint so one noise
verdict suppresses a flappy alert permanently. Blips then flow through
the same title/noise dedup and triage as live alerts; the prompt
explains their semantics (self-resolved; judge the underlying cause;
null-routed per the repo route config = noise).
Verified against live data: currently-firing KubeCPUOvercommit
excluded, injected KubeHpaMaxedOut blip synthesized correctly;
sha256sum confirmed present in ci-claude; lookback query tested via
the service proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Owner rule: ALL guest lifecycle goes through tofu; no manual pct/qm
destroy, by anyone. The rustfs trio (800/801/802) was never
tofu-managed, so the #265 sweep could not reach it. Stage 1 adopts the
stopped containers into state (pipeline TOFU_IMPORT hook, commands in
the file header); stage 2 deletes this file and the apply destroys
them.
lifecycle ignore_changes=all + started=false keep the shims inert:
tofu must never reconcile or start these, only own their destruction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Enables the single-click HA onboarding shipped in shelly-manager PR #41:
shelly-manager registers a provisioned Shelly into HA's native integration
by driving HA's config-flow + device-registry API.
- shelly-manager-llat ExternalSecret projects the full-admin HA LLAT from
OpenBao (apps/home-assistant/shelly-manager-llat, key 'token').
- HomeAssistant__BaseUrl = http://home-assistant (in-namespace svc :80 → nginx
sidecar → HA :8123; nginx forwards the WS upgrade, so both the REST config-
flow and the device-registry websocket work). Reachability confirmed: same-
namespace ingress is allowed by the default-deny-ingress CNP's home-assistant
rule (probed :80 → 401 = reachable).
- HomeAssistant__Token from the new secret.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #259 lesson, both directions: the agent rightly refused
unauthenticated instructions embedded in the issue body — and then had
no channel for the owner's authentic reply 18 minutes later, so a
human ended up doing the sweep the agent should have owned.
- Pick step: an issue with an alert-fix:assessed marker becomes
eligible again when a TRUSTED_AUTHORS comment is NEWER than the
latest marker (timestamps compared in jq — busybox ash test has no
string-<). Owner replies on the issue now drive the loop.
- Trusted comments are appended to the agent's issue.md as 'Verified
owner guidance'. Trust anchor is the hardcoded author list in this
file — deliberately not the Gitea permission API, because the
pipeline itself comments as gitea_admin and agent-authored text must
never be able to masquerade as owner guidance.
- Prompt: verified guidance is authoritative for intent and scope (may
lift the surgical ceiling when it calls for a sweep), does not lift
the edit-path allowlist, and factual mismatches with live state are
surfaced, not silently resolved in either direction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Owner directive: PVE runs ONLY the three k8s VMs. The proxmox-dns trio
(lxc 118/218/318), rustfs trio (lxc 800/801/802), fortress (vm 102)
and paperless (vm 111) are decommissioned; guest definitions are being
destroyed on PVE out-of-band.
Removed:
- opentofu: lxc_118_proxmox_dns.tf, vm_102_fortress.tf,
vm_111_paperless.tf, backup_jobs.tf (the vzdump job covered ONLY
destroyed guests — next tofu apply removes the job from PVE; k8s
data protection is Velero/CNPG)
- ansible: roles/proxmox-dns, install-proxmox-dns{,-ha}.yml,
check-chaos-testing.yml (tested failover of decommissioned DNS-HA
and postgres LXCs), proxmox_dns_vault.yml, rustfs_vault.yml,
[proxmox_dns_ha] inventory group
- monitoring: dns-servers scrape job in both the k8s HelmRelease and
the legacy prometheus-server template — resolves TargetDown
(dns-servers)
Closes#259
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Woodpecker's k8s backend deletes each workspace PVC when a pipeline
ends, but ceph-filesystem has reclaimPolicy Retain, so every finished
pipeline left a Released PV pinning a CephFS subvolume — 191 of them
by 2026-07-16 (found in #262). New ceph-filesystem-ephemeral class
(identical parameters, reclaimPolicy Delete) used only by the CI
workspace volumes; production PVCs keep the undoable Retain default.
Existing Released woodpecker-ci PVs are swept out-of-band (they
predate this class and are unaffected by it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
- obs checks: run e2e-helper in the monitoring namespace — default is
not on monitoring's default-deny-ingress allow-list, so every
loki/prometheus/alertmanager check timed out (curl exit 28)
- node count: 4 -> 3 (k8s-1/2/3; there is no 4th node)
- CNPG exec targets: resolve the primary via
cnpg.io/cluster + cnpg.io/instanceRole=primary labels instead of
hardcoding pod names (pprl-pg-1 no longer exists; ak-pg-1 was one
failover away from the same break)
- no-failing-pods: exclude the woodpecker-ci namespace — the check
raced the e2e run's own runner pod (ContainerCreating, 4s old)
- purge_released_pvs: drop jq (not in ci-k8s image — the function has
been a silent no-op, 305 Released PVs currently pinned); jsonpath +
awk instead
Closes#262
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
VMM has no cloud-init and only makes blank disks, so replace the cloud-init VM
step with the real method: remastered netinst ISO auto-booting a preseed that
powers off at end (breaks the guest-reboot->CD reinstall loop). Debian 13, DHCP,
Default VM Network. Document the Netbird mesh-reachability caveat (CI/home not
peers) and the obg.renner.uno split-horizon DNS trap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- inventory: claude-ws at its current DHCP lease (10.3.40.78). Switch to the
stable Netbird mesh IP post-enrolment; the obg.renner.uno FQDN is
split-horizon and resolves to the public proxy from home, so it must not be
used as ansible_host.
- group_vars/claude_workstation/vault.yml: vault'd netbird_setup_key, scoped to
the group so install-netbird-client.yml only enrols this host.
- .woodpecker/deploy-netbird-client.yml: pipeline running
install-netbird-client.yml --limit claude_workstation (manual + master-path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
reload_all does not re-read YAML-mode Lovelace dashboards (HA caches them until
restart or a lovelace/config force=true WS call), so merged dashboard changes
never appeared without a manual force-reload.
After a pull-merge that touches dashboards/ or ui-lovelace.yaml, config-sync now
also calls shell_command.reload_lovelace_yaml (defined in the HA config repo,
packages/lovelace_reload.yaml) which runs a websockets helper inside the HA
container to force the re-read. busybox can't speak WebSocket, hence the hand-off.
Needs a one-time HA restart to register the shell_command domain (not covered by
reload_all); until then reload_lovelace logs a warning and sync is otherwise fine.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
Turns on the app's new Auth:ReverseProxy mode (shelly-manager PR #40) so
the X-authentik-username header injected by the proxy outpost signs the
gate session straight into the app — removing the double login introduced
with the auth hardening. Trust is scoped to the pod CIDR because the
outpost pod address is dynamic; the namespace CNP remains the control
that keeps other pods off the app port. Inert until the image built from
shelly-manager PR #40 is promoted (unknown env keys are ignored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TargetDown carries no namespace label, so two simultaneous TargetDowns
(job=pve and job=dns-servers) collapsed into one '[alert] TargetDown
(cluster)' issue — the dns-servers root cause was never triaged
because the title dedup buried it under the pve issue. (Prometheus
shows 'context deadline exceeded' on all three dns-servers targets
since 2026-07-13: names resolve, the :12345 metrics endpoint doesn't
answer — cause to be triaged once the issue exists.)
Title key is now namespace // job // "cluster". Verified against the
live API: yields distinct '(pve)' and '(dns-servers)' titles.
Migration: retitle open issue #250 to '[alert] TargetDown (pve)' after
merge so the pve thread keeps deduping; the monitor then files a fresh
dns-servers issue for the fix agent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The fix agent's #252 assessment surfaced this: KubeCPUOvercommit is
deliberately routed to receiver 'null' in the Alertmanager config
('by design unsatisfiable on a small homelab cluster'), yet the
triage pipeline filed an issue for it — re-litigating a decision the
owner already made in the route config. The Alertmanager v2 alerts
API returns each alert's matched receivers, so drop alerts whose
receivers are all 'null' before triage. Filter verified against the
live API: KubeCPUOvercommit and Watchdog drop, the email-routed
alerts stay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
No pipeline passed --model, so runs floated on the ci-claude image's
CLI default — behavior could shift silently on the next image bump.
All four headless agents (alert-triage, issue-fix, security-fix,
update-audit) now pin claude-sonnet-5 explicitly; upgrade per-agent
by editing the flag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Found on the first real use of the restricted identity: assessing the
VeleroBackupFailed issue (#251) needs backups.velero.io (latest
apps-daily run is PartiallyFailed — data-mover pod unschedulable on an
unbound PVC), and the ClusterRole had neither velero.io nor CNPG.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:
- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
ci-diagnostics with a read-only ClusterRole (workloads, events,
pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
plus a namespaced services/proxy Role for the Prometheus and
Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
the admin kubeconfig (no new long-lived secret), builds
.ci-diag/kubeconfig, copies the static kubectl binary for the
jq-only ci-claude image, smoke-tests with the restricted identity,
and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
hypothesis you could have settled with one read-only command but
didn't is a defect in your output'; fix agent must test the issue's
claims before building on them; minimal-quote rule against dumping
logs into public issues.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The first issue-fix run on #249 asserted the openbao pod was '0/1 and
sealed' — it was Ready and unsealed the whole time; the alert fired
because the chart's updateStrategy: OnDelete holds the 2.5.5→2.6.0
image bump until an operator deletes the pod. The agent has no cluster
access and invented runtime state from the alert's generic annotation
text.
- issue-fix prompt: runtime claims must be attributed or framed as
hypothesis, with read-only verification commands in the summary.
- alert-monitor prompt: when context.txt does not confirm the alert's
symptom, say so explicitly — the mismatch is often the real clue.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Two new cron pipelines closing the loop from firing alert to fix PR:
- alert-monitor (*/15 min): reads active alerts from Alertmanager via the
apiserver service proxy (woodpecker-ci is netpol-blocked from monitoring
ClusterIPs; the Authentik gate blocks the public hostname), dedups
against open issues and a persistent noise-log issue, then a headless
Claude agent triages the new ones against the repo plus a cluster
snapshot. Confirmed problems become Gitea issues; noise is recorded in
the noise log so it is never re-triaged.
- issue-fix (hourly): picks the oldest open '[alert] ...' issue with no
fix branch and no assessed-marker comment, hands it to a fix agent
(same tool sandbox as security-fix), pushes alert-fix/issue-<n> and
opens a PR with 'Closes #<n>'. Not-repo-fixable verdicts are posted as
an issue comment instead.
- e2e.yml: scope the bare 'event: cron' to the three existing cron names
— otherwise the 15-minute cron would piggyback a full e2e run every 15
minutes.
The ci-k8s image has no jq, so the alert fetch step only talks to the
cluster; all JSON filtering/dedup/gating lives in the ci-claude step.
Cron registration (after this lands on master):
woodpecker repo cron add --repo gitea_admin/homelab \
--name alert-monitor --branch master --schedule '*/15 * * * *'
woodpecker repo cron add --repo gitea_admin/homelab \
--name issue-fix --branch master --schedule '41 * * * *'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The public-release auth work (shelly-manager PR #38) makes the app refuse
to start without ADMIN_PASSWORD; Flux promoted the new image and the pod
went CrashLoopBackOff. Adds the shelly-manager-auth ExternalSecret
(OpenBao: apps/home-assistant/shelly-manager-auth) and the env wiring,
repoints both probes to the new unauthenticated /health endpoint (closes
the long-standing probe gap — / would now 302 to /login), and updates the
stale 'app has no auth of its own' annotation: the macvlan bind pinning
stays as defense in depth.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Der Configurator-Pod hing (UI nicht erreichbar, "No Room Selected" beim
Zonen-Editieren, Live-Tracking eingefroren): RSS lag bei 254Mi/256Mi und
~0,95 CPU-Kerne – klassisches Node/V8 GC-Thrashing am cgroup-Memory-Limit.
Der Prozess blieb knapp unter dem Limit, wurde also nie OOMKilled und kam
nie von selbst zurück; ein Neustart hat ihn befreit (frisch ~115Mi).
- limits.memory 256Mi → 512Mi (Headroom gegen GC-Thrash), requests.memory
64Mi → 128Mi (realistischere Baseline).
- Liveness/Readiness httpGet auf / (Port http). Bisher gab es keine Probe,
daher hat kubelet den hängenden Server nie neu gestartet. Jetzt schlägt
die Probe beim Einfrieren fehl und der Pod wird automatisch recycelt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Der Everything-Presence mmWave Zone-Configurator lief bereits im Cluster
(Deployment in ns home-assistant, UI :42069), war aber nur ClusterIP-intern
erreichbar. Er hält ein HA Long-Lived Token und kann die ESPHome-Zonen-Entities
aller Präsenzmelder umschreiben, hat aber keine eigene Auth — daher wird er
(wie vscode/shelly-manager/esphome) hinter ein Authentik Passkey-Gate gelegt,
nicht nackt exponiert.
Neu:
- Authentik-Blueprint zone-configurator-gate: Group "Zone Configurators"
(k.renner), ProxyProvider (passkey-stepup, internal_host …:42069,
external_host zone-configurator.home.renner.uno), Application, PolicyBinding,
dedizierter Outpost.
- ExternalName-Service ak-outpost-zone-configurator-gate + Traefik-Ingress
(cert-manager letsencrypt-cluster) im app-Verzeichnis.
- Cilium-Egress (authentik) + Ingress-Allow (home-assistant-ns) für
outpost -> zone-configurator:42069. Port 42069 ist eindeutig (wie esphome
:6052), daher kein Port-Overlap mit anderen Apps.
Traffic: browser -> traefik -> ak-outpost-zone-configurator-gate -> Outpost
(Passkey) -> zone-configurator.svc:42069.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The woodpecker-ci namespace enforces pod-security `privileged` (required for
the allow-listed docker/buildx plugin), Woodpecker runs WOODPECKER_OPEN=true,
and no admission controller exists. PodSecurity is all-or-nothing per
namespace, so the buildx exemption also un-gates hostPath/hostNetwork/hostPID/
hostIPC and privileged on every step pod a pipeline author can shape.
Reduce the blast radius with three defense-in-depth controls, none of which
weaken the actual build capability (this repo's builds use rootless kaniko;
only the external buildx plugin needs privileged, and it stays allow-listed):
- ValidatingAdmissionPolicy (GA/in-tree on k8s v1.35, no new controller):
denies host namespaces + hostPath for all step pods, and privileged for
every image except woodpeckerci/plugin-docker-buildx. Re-narrows the
PodSecurity exemption at the authoritative K8s admission layer. CEL verified
to compile via server-side dry-run.
- CiliumNetworkPolicy: ingress default-deny for step pods (house style,
egress left default-allow so builds keep reaching the internet + Gitea).
- ResourceQuota + LimitRange: bound pods/CPU/memory/PVCs against a
resource-exhaustion DoS; sized generously so current builds are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
status.renner.uno / status.k.renner.uno is a public, unauthenticated gatus
page. Its endpoints named the internal service estate (Authentik, Gitea,
Immich, Grafana, HA, Paperless, Proxmox, PBS, UDM DNS, Mosquitto) and gatus
exposed each check's target hostname/URL/port — and on failure the error
text (e.g. mosquitto.home-assistant.svc.cluster.local, 10.2.40.1) — via the
public results and /api/v1/endpoints/statuses API (all hide-* default false).
Per owner decision keep the page public (not Authentik-gated) but scrub the
disclosure:
- rename every endpoint to an opaque label (Web Service N / Infrastructure N),
removing product/vendor/hostname identifiers from the public `name`
- add a per-endpoint ui block with hide-hostname/hide-url/hide-port/
hide-conditions/hide-errors: true so gatus scrubs targets server-side
All checks keep running unchanged (real targets stay in `url`, visible only
to git/cluster operators). Public page now shows group + opaque label +
up/down + response time only. Maintainer opaque->real map kept as a comment.
Fixes#206
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CI automation keys were deployed to root's (and ladmin's) authorized_keys
on every host with no key_options, giving any holder of a CI key unrestricted
root fleet-wide (issue #184).
- Restrict woodpecker-ci@homelab with from="10.244.0.0/16,fd10:244::/48" — a
network-of-origin restriction to the cluster pod CIDRs, so the key only
authenticates from within the CI's network segment. Determined by inspecting
Cilium: routing-mode=tunnel with enable-ipv4-masquerade=true but
ipv4-native-routing-cidr=10.0.0.0/8 (and v6 fd00::/8), which covers every
10.2.x / fd.. LAN SSH target, so pod egress to the fleet is NOT SNAT'd to the
node IP — pods present their real POD IP, BGP-advertised to the UDM
(cilium-bgp.yaml). The stable boundary is therefore the pod subnet
(talos cluster.yaml podSubnets: 10.244.0.0/16 + fd10:244::/48), which
survives node DHCP-lease churn and pod rescheduling across k8s-1/2/3 — unlike
pinning any node's current (DHCP) IP. Value lives in group_vars/all/vars.yml
(ci_ssh_from_cidrs) as the single source of truth.
- Remove ansible-runner@ci-new entirely from the play, the common role, and
common_ssh_public_keys — dead key, no such host exists anywhere in the repo.
- Keep krenner@fortress (human) unrestricted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The velero-bootstrap-restore Job (restore-job.yaml + its RBAC) was never
wired into any Flux Kustomization — no `path` field in
kubernetes/clusters/homelab/ referenced it, and the parent
infrastructure/kustomization.yaml only aggregates `velero/`, not
`velero-bootstrap-restore/`. It never ran, creating false confidence that
app-data auto-restore-on-bootstrap existed.
Per owner decision on #200: delete it and rely on the documented manual
restore procedure in kubernetes/RUNBOOK-cold-rebuild.md. The real seed
path is the seed-* Kustomizations (rsync from legacy host) plus the
per-app restore steps in the runbook.
Also updates two now-dangling doc comments in
post-crds/velero-schedules.yaml that referenced the deleted Job's
SCHEDULES mechanism, pointing them at the runbook instead.
kubectl kustomize on infrastructure/ and infrastructure/post-crds/ both
build clean after removal.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flux pulls manifests from the in-cluster Gitea, which is itself Flux-managed
(Ceph data, in-cluster DNS+Traefik), so a cold rebuild has no entry point.
Give it one:
- bootstrap.sh: FLUX_SOURCE=github selects 'flux bootstrap github' against the
mirror (github.com/konairius/Homelab, branch master) instead of
'flux bootstrap gitea'; defaults to gitea so steady-state is unchanged.
- kubernetes/RUNBOOK-cold-rebuild.md: single tofu->talos->flux->openbao->apps
runbook, with the break-glass github bootstrap and the manual OpenBao
snapshot-restore+unseal step wired in.
The GitHub mirror is already kept current by Gitea's own native push-mirror
(server-side, sync_on_commit=true) — no setup and no new credential required;
the runbook just references it. OpenBao restore stays manual (unseal key out
of CI, in pass).
Refs #187
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR #231 scoped the openbao ClusterSecretStore to the 12 namespaces that
had ExternalSecrets at the time. PR #236 then added a new consumer in the
crowdsec namespace (the bouncer key), which the selector refused
(SecretSyncedError). Add crowdsec to the allowed namespaces.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The apps namespace (Gitea, Woodpecker, gatus) had no Velero schedule at
all, so the in-cluster Gitea (data-gitea-0, the Flux GitOps source of
truth) and Woodpecker (data-woodpecker-server-0) PVCs migrated from the
legacy LXCs were unprotected. The only PBS coverage (opentofu/backup_jobs.tf
VMID 114/231) targets the original LXCs, not the live cluster PVCs.
Add an apps-daily Schedule (06:45 local, next free stagger slot) mirroring
the existing PVC-only pattern; no CNPG cluster runs in this namespace so no
labelSelector exclusion is needed. Also register it in the bootstrap-restore
Job's SCHEDULES list so a fresh-cluster bootstrap restores it like the
others.
Fixes#186
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The Talos bootstrap inline manifest (patches/cilium-cni.yaml) and its
standalone copy (cilium.yaml) embedded Secret/cilium-ca (RSA ca.key) and
Secret/hubble-server-certs (RSA tls.key) as plaintext base64 — rendered
Helm output from hubble.tls.auto.method=helm. Anyone with repo/history
read access could mint certs under the Cilium CA and impersonate/decrypt
the Hubble mTLS channel.
Switch to hubble.tls.auto.method=cronJob: replace the two committed-key
Secrets with the certgen ServiceAccount/Role/RoleBinding/Job/CronJob.
The Job generates (and the CronJob renews) Secret cilium-ca and issues
Secret/hubble-server-certs in-cluster at install. No private key material
is committed. Verified with `helm template cilium/cilium --version 1.19.2`:
the only delta between the two methods is exactly these 7 resources; the
ConfigMap and cilium-agent DaemonSet are unchanged, and the agent's
hubble-tls volume is already optional:true so it starts before certgen
runs. Both files and the inline block scalar validated as YAML.
This is the code half of the fix. Rotating the already-leaked live CA +
Hubble certs and scrubbing them from git history is an operator task
tracked on the issue.
Refs #188
Owner confirmed on #225 that all five roles flagged in #198 are legacy and
were decommissioned some time ago; the earlier inventory-presence liveness
check was wrong (inventory has stale entries for services migrated into the
k8s cluster). So instead of sanitizing committed secrets, remove the dead
code that held them.
Deleted role directories (all owner-confirmed legacy):
- ansible/roles/immich -> now runs in-cluster (kubernetes/apps/immich, secrets via OpenBao ESO)
- ansible/roles/grafana-observability -> dashboards now live in kubernetes/apps/monitoring/dashboards
- ansible/roles/postgres-simple-ha
- ansible/roles/keepalived-ntp
- ansible/roles/keepalived-vip
Deleted plays that exclusively orchestrate the deleted legacy roles:
- ansible/plays/install-immich.yml
- ansible/plays/configure-grafana-observability.yml
- ansible/plays/backup-immich-db.yml (one-time migration dump from legacy LXC 225)
- ansible/plays/configure-immich-db-password.yml (loads the deleted immich role)
Edited (kept, live):
- ansible/plays/update.yml -> dropped the `import_playbook: install-immich.yml`
line so the CI update path (.woodpecker/update-all.yml -> plays/update.yml)
keeps working.
NOT touched (ambiguous — reference companion roles the owner did NOT list as
legacy; flagged on #225 for owner decision, no dangling roles deleted here):
- install-ntp-ha.yml (also uses chrony-ntp-server)
- install-proxmox-dns-ha.yml (also uses proxmox-dns, which is still live)
- install-authentik-ha.yml (also uses authentik-stateless)
These three plays now reference deleted keepalived-*/postgres-simple-ha roles;
their companion roles + inventory groups need an owner call before removal.
Refs #198, #225. Coordinates with #208 (decommission sweep) for immich.
Closes gitea issue #197. The update-audit agent fetches untrusted
third-party release pages via WebFetch/WebSearch and its Write output
is posted verbatim to a public Gitea issue, so injected content in a
fetched page could try to redirect the agent's behavior. Harden the
tool surface (verified empirically against the pinned Claude Code CLI):
- Drop WebSearch: it cannot be domain-scoped like WebFetch, and the
prompt already gives explicit source URLs or a well-known registry
for everything it needs to look up.
- Add ci/agents/update-audit/settings.json (loaded via --settings) that
denies Read on /proc, /root, /home, /etc, and the repo's own .git/ —
closing the credential-file and env-var-dump read paths a
prompt-injected instruction could otherwise try. Verified this deny
mechanism actually blocks reads, unlike Write path-scoping, which
was tested and found NOT enforced under --permission-mode acceptEdits
(documented in the pipeline comment; a full WebFetch domain allowlist
was also considered and rejected, since Helm chart lookups legitimately
hit whichever domain each HelmRelease's own trusted source URL points
at).
- prompt.md now tells the agent fetched content is untrusted data, not
instructions, and to never echo it verbatim into the report.
Not fixed here: GITEA_TOKEN is still the broad registry/release-writing
token (no write:issue-only token exists yet) — that requires an
operator to mint one; tracked via a comment on #197 and a code comment
at the secret binding.
PR #235 (fix/issue-193-woodpecker-ci-hardening) already adds
kubernetes/infrastructure/network-policies/woodpecker-ci-ns.yaml, with a
deliberately different (and better-informed) egress design: default-allow,
because a past attempt at egress enforcement in this namespace broke
crowdsec (66dbe9f) — history this branch didn't have context on.
PR #231 (fix/issue-202-openbao-hardening) already adds an egress block to
openbao-ns.yaml as part of a broader OpenBao hardening pass, including a
"world" allowance (OIDC) this branch's version omitted.
Dropping both here to avoid shipping a conflicting/duplicate policy;
external-secrets-ns.yaml (this issue's third gap, not covered by any other
open PR) is the only change left on this branch.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
bouncer-key.yaml committed the CrowdSec<->ingress-nginx bouncer key in
cleartext across two namespaces, outside the ESO/OpenBao pattern used
elsewhere in the cluster. Replace both Secrets with ExternalSecrets
pulling the same value from a single OpenBao KV path
(infrastructure/crowdsec-bouncer-key), matching the existing
apps/infrastructure ExternalSecret convention.
Requires the owner to populate OpenBao before merge:
bao kv put -mount=homelab infrastructure/crowdsec-bouncer-key key=<value>
Fixes#212
Closes the gaps from #194: external-secrets and woodpecker-ci had no
CiliumNetworkPolicy at all (Cilium defaults to allow-all absent one), and
openbao-ns.yaml had an ingress-only CNP with no egress block, which leaves
egress from openbao pods completely unenforced rather than merely permissive.
- external-secrets-ns.yaml: default-deny-ingress; egress scoped to the
namespace, DNS, and the openbao KV API on :8200 (the only
ClusterSecretStore in this cluster) plus host/remote-node for
kube-apiserver (webhook admission, Secret watch/create, leader election).
- woodpecker-ci-ns.yaml: this namespace hosts only ephemeral, sometimes
privileged, pipeline-defined step pods (no static workloads) — the goal
is blocking lateral movement into unrelated namespaces while still
allowing the git clone (east-west via k8s_gateway -> traefik, since
gitea.renner.uno now has an in-cluster Ingress) and the broad internet
egress every CI system legitimately needs (image pulls, package
managers).
- openbao-ns.yaml: added an egress block (DNS, intra-namespace, host/
remote-node for the k8s auth method + kubelet) so egress enforcement is
actually turned on, not just missing.
music-assistant is deliberately NOT covered here: its only workload runs
hostNetwork: true, so a namespace-scoped CiliumNetworkPolicy would select
no enforceable pod-network endpoint and be a no-op (same documented
limitation as the rook-ceph Ceph daemons). Left for a comment on #194
rather than shipping a policy that looks protective but isn't.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #209. All four ci/images/*/Containerfile FROM lines were
tag-pinned only (alpine:3.21, node:22-alpine), letting the base silently
change between builds. Pin each to its current manifest-list digest.
ci/images/k8s/Containerfile downloaded talosctl, kubectl, and velero via
curl with no integrity check, and installed flux via `curl | bash`
(itself an unpinned, unverified fetch-and-execute). Replace the flux
install with a pinned-version tarball download verified against its
published sha256 checksum, same as talosctl/kubectl/velero now are.
kubectl's version was also resolved dynamically from dl.k8s.io's
stable.txt at build time instead of being pinned; add KUBECTL_VERSION.
ci/images/tofu/Containerfile's tofu.zip download gets the same
sha256 verification.
ci/images/claude/Containerfile's CLAUDE_CODE_VERSION defaulted to
`latest` and was never overridden by build-ci-images.yml, so the image
holding repo/issue-write tokens floated to whatever npm resolved at
build time. Pin it to the current release (2.1.209).
Checksums and digests verified against the respective registries/release
APIs at fix time (2026-07-14).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Adds minAvailable:1 PDBs for the data-bearing single-replica StatefulSets
that had no eviction protection: gitea, woodpecker-server, and paperless's
protonmail-bridge + syncthing sidecars. On a 1-replica workload this makes
the eviction API refuse *voluntary* evictions, so a node drain blocks and
surfaces the pod instead of yanking a live SQLite DB / RWO-PVC pod mid-write.
Involuntary disruptions (node crash, OOM) are unaffected.
Deliberately excluded, with rationale:
- CNPG clusters im-pg and gf-pg (instances:1): CloudNativePG intentionally
omits a primary PDB for single-instance clusters (no failover target), and
handles graceful primary shutdown itself. A manual maxUnavailable:0 would
only wedge node drains with no HA benefit.
- Ceph mons: the Rook operator self-manages quorum-preserving mon/OSD PDBs
(disruptionManagement.managePodBudgets default). A hand-written PDB would
be redundant or fight the operator on each reconcile.
- Stateless single-replica Deployments (redis/tika/gotenberg caches, etc.):
a blocking PDB gives no recoverability benefit, only blocks drains.
Refs #201
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The certificates role fetched https://authentik.renner.uno with
validate_certs: false and fed whatever it got back into
update-ca-certificates on every Ansible-managed host — an on-path/DNS
attacker during that fetch could plant a rogue CA cluster-wide.
authentik.renner.uno also turned out to be a stale hostname: it isn't
routed by the current Traefik/cert-manager ingress. The live service
(auth.renner.uno) already gets a publicly-trusted Let's Encrypt cert via
cert-manager, which the OS default CA bundle already trusts. Nothing else
in the repo depends on the installed authentik.crt, so removing the task
(and its now-unused handler) closes the hole with no functional loss.
Fixes#185
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Issue #210: neither the SOPS age key nor ANSIBLE_VAULT_PASSWORD has a
documented offline backup location. The age-key archive was previously
bundled into the Phase 4 point-of-no-return checklist, blocking it on the
reconcile/unseal soak for no reason — backing it up to pass is safe to do
immediately since it only adds a copy. Also documents a pass location and
verification procedure for the Ansible Vault password, which gates PBS and
other Ansible-Vault-encrypted credentials and had no backup plan at all.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Closes#205. The websecure entrypoint had zero response hardening and no
rate limiting — CrowdSec is detection-only since the 2026-06-27 IPv6
mass-ban outage, and no headers/rate-limit Middleware existed anywhere in
the cluster, so logins sat behind app-level throttling only.
Adds two Middlewares wired as entrypoint defaults via the same
additionalArguments mechanism the (still-detached) crowdsec-bouncer
Middleware documents for its own re-enable:
- security-headers: HSTS, nosniff, xss-filter, referrer-policy,
permissions-policy, and a conservative frame-ancestors 'self' CSP
(deliberately not a full default-src policy — apps here span Paperless,
Immich, Grafana, Gitea, Home Assistant, etc. with very different
script/style/font sources).
- rate-limit: generous defaults (100 req/s, burst 200) as a baseline
abuse/brute-force backstop rather than traffic shaping, keyed off the
real client IP via sourceCriterion.ipStrategy.excludedIPs (same trusted
pod CIDRs as forwardedHeaders/crowdsec-bouncer) so it doesn't collapse
every external client onto one shared budget — the same client-IP-
visibility gap that caused the CrowdSec outage this fix is otherwise
independent of.
CrowdSec enforcement itself stays disabled; that re-enable is separate,
documented, unaddressed work.
Addresses the safely-codeable items of the #202 hardening bundle:
- snapshot-pvc: add kustomize.toolkit.fluxcd.io/prune: disabled so the sole
supported restore artifact (raft snapshots PVC) survives a manifest
rename/move, matching sibling data PVCs.
- policies/eso-read.hcl: version-control the eso-read policy that only lived
as a heredoc in the migration doc + live config; reference-only (OpenBao
policies aren't Flux-reconciled), with apply/verify commands inline.
- clustersecretstore: add conditions restricting the store to the exact
current consumer namespaces, closing the "any namespace can read all of
homelab/*" gap.
- network-policies/openbao-ns: add an egress stanza (default-deny egress,
allowing intra-ns, kube-dns, kube-apiserver/host, world, ICMP) to block
lateral in-cluster egress from the secrets backend.
Listener TLS/mTLS and per-app KV path re-segregation are the disruptive
remaining items and are raised as owner decisions on #202. Single-replica
raft is a documented/accepted tradeoff, not touched.
Fixes#202
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gitea (LXC 114), the woodpecker server (231) and the woodpecker agents
(910/911) were migrated into the cluster (Phase 2: Service flipped
ExternalName -> ClusterIP, now in-cluster StatefulSets/Deployment; see
kubernetes/apps/MIGRATION-gitea-woodpecker.md) and the Proxmox guests
decommissioned -- but their OpenTofu declarations were left behind with
onboot/start = true. An unscoped `tofu apply` (deploy-infrastructure.yml)
would attempt to recreate the retired stack, and backup_jobs.tf still
advertised the dead VMIDs as PBS-protected.
- delete opentofu/woodpecker_agents.tf, lxc_231_woodpecker.tf,
lxc_114_gitea_srv_renner_uno.tf
- drop VMIDs 114/231 and the 910.. agent loop from backup_jobs.tf
- remove the now-orphaned woodpecker_agents local + woodpecker_agent_count
variable (only those files referenced them)
- delete the dead .woodpecker/deploy-woodpecker-agent.yml and
install-woodpecker-server.yml pipelines (their ansible plays/roles no
longer exist)
REQUIRES `tofu state rm` of the four module resources BEFORE this merges
/ any apply runs, otherwise the plan destroy-plans them. See PR body.
Refs #199
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
object_storage.tf provisioned all four Hetzner buckets (CNPG WAL/base
backups, Velero PVC backups, cluster bootstrap secrets, tfstate) with
object_locking = false and no versioning, so a single leaked/misused S3
credential could irrecoverably delete backups and state.
Add minio_s3_bucket_versioning for all four buckets: a plain
DeleteObject no longer destroys data, the prior version survives until
purged by version ID.
object_locking is NOT flipped to true here: per the aminueza/minio
provider source, object_locking is only consumed on bucket Create, has
no Update handling and no ForceNew, so setting it on these
already-live buckets would be silently ignored -- not a real fix.
Enabling it requires destroying and recreating each bucket (full data
migration), which needs an operator decision; left as a follow-up
(see gitea issue #191 comments).
Also add minio_s3_bucket_lifecycle noncurrent-version expiration
(30 days) on the two high-churn backup buckets (cnpg, velero) to match
the retention window already enforced at the app layer (CNPG Barman
ObjectStore retentionPolicy, Velero Schedule ttl), so old S3 versions
don't accumulate indefinitely. secrets/tfstate buckets are low-churn
and left uncapped.
Fixes#191
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The three backed-up CNPG clusters (ak-pg/im-pg/pprl-pg) bootstrap via
bootstrap.initdb, so a cold rebuild silently comes up with EMPTY databases:
barman WAL+base backups archive to Hetzner but are never wired back into a
rebuild, and the restore path was undocumented (issue #190).
- Add kubernetes/seed/RECOVERY.md: a tested manual recovery runbook with
per-cluster bootstrap.recovery + externalClusters blocks, precondition
checks, verify/revert steps, and a non-destructive CNPG restore-test drill.
- Explain the safety model: CNPG bootstrap is inert on a live cluster, so an
initdb->recovery swap cannot clobber live data; recovery has no initdb
fallback (hangs on an empty bucket), which is why permanent recovery is an
owner decision, not flipped here.
- Fix the seed-authentik Flux KS comment that falsely claimed rebuilds
auto-recover from Garage (decommissioned; backups are on Hetzner now).
- Point seed-immich / seed-paperless KS comments at the runbook.
Refs #190
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
netbird-ingress.yaml's ImagePolicy used ">=0.73.0" — unbounded above,
unlike every other of the 23 enrolled policies which pin to a major
(or major.minor). Since image-automation pushes bumps straight to
master with no PR gate, an unbounded range lets any future upstream
release (including a breaking 1.x) land automatically.
Bound it to "0.x", consistent with cnpg-barman-plugin's precedent for
pre-1.0 images: the currently deployed v0.74.4-rootless stays in
range, and a 1.x jump now requires a manual policy widen. Also adds
netbird-ingress to the README's enrolled-policies table, where it was
missing despite being wired into the kustomization.
Fixes#204 (partial — see issue comment for the scoped-token and
PR-gating items, which need an owner decision).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
remove_ceph_cpool.yml (fleet-wide zfs destroy on pve_cluster) and
cleanup-gitea-mess.yml (deletes gitea service/binary/config/user on
all:!gitea.srv.renner.uno) are archived but still directly runnable
with no confirmation guard. Add the same fail-task safety-switch
pattern already used in plays/update-dist-upgrade.yml, requiring
-e 'confirm_destroy=true' before either play executes.
Fixes#207
Issue #195 flagged KasmWeb (browser.renner.uno) for having no Authentik
gate and hardcoded weak credentials. Owner comment overrides the
originally-proposed auth-gate fix: "can be decomissiond". Removes the
app dir, its apps/kustomization.yaml entry, the orphaned firefox
image-automation policy, and updates comments in the shared
insecure-backend ServersTransport and external-dns-unifi HelmRelease
that referenced test-browser as a consumer/example.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
home-assistant, esphome, shelly-manager, matter-server, otbr, whisper,
speech-to-phrase, and zone-configurator all deploy into the
home-assistant namespace (per their Kustomization's `namespace:`
field, not their kubernetes/apps/ dir), and none of their PVCs were
covered by any Velero schedule -- including home-assistant-config,
which holds the .storage entity registry, recorder DB, and
secrets.yaml. Adds home-assistant-daily at the next free 06:45
stagger slot, PVC-scoped like the existing schedules.
Fixes#192
Confirmed-dead, low-risk removals only. Ambiguous items (ansible immich
role, ansible-side proxmox-dns wiring) are left in place pending owner
confirmation; tofu LXC drift is owned by #199.
- cve_list.txt: orphaned agent output (added in f22927b), referenced by
nothing; security-scan.yml uses trivy JSON outputs, not this file.
- .woodpecker/deploy-woodpecker-agent.yml: calls
ansible/plays/install-woodpecker-agent.yml and role woodpecker-agent/,
both of which no longer exist; push-trigger paths also gone.
- .woodpecker/install-woodpecker-server.yml: calls
ansible/plays/install-woodpecker-server.yml, which no longer exists.
- kubernetes/apps/woodpecker/server.yaml: fix stale comment claiming the
agent is replicas:0 (phase 2) with LXC agents still building —
contradicted by agent.yaml replicas:1.
Fixes#208 (partial)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy-infrastructure.yml's plan step ran tofu plan (which evaluates
provider/data-source code) on push to any branch, with proxmox/hetzner/unifi
secrets mounted — a malicious .tf on a feature branch could exfiltrate them
pre-merge. deploy-netbird-server.yml was missing the branch: master gate its
siblings (deploy-netbird-proxy.yml, deploy-woodpecker-agent.yml) already
have, so a push to any branch touching netbird-server files triggered a real
ansible deploy with SSH key, vault pass, and proxmox token.
Fixes#196
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
shelly-manager will build+flash BLE proxies by spawning short-lived
esphome Jobs (on the shelly-manager-iot macvlan → mDNS → OTA:3232) and
streaming their logs. Give it a namespace-scoped identity to do so:
- ServiceAccount shelly-manager + Role shelly-manager-firmware
(jobs: create/get/list/watch/delete; pods: get/list/watch;
pods/log: get) + RoleBinding. No cluster-wide, secret, or exec rights.
- Wire serviceAccountName into the Deployment.
- Sync esphome/shelly-ble-proxy.yaml logger to the lean level already
OTA'd to the device (esp32_ble_client DEBUG, not VERBOSE) so the PVC
stops filling; VERBOSE is now an on-demand diagnostic via the pipeline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Provisioning a Shelly over BLE fails at the GATT connect and the cause is
inside the ESP's BLE stack — invisible from shelly-manager's side. To
diagnose and iterate without a USB re-flash per change:
* ota: (esphome platform) so future config changes push over WiFi from the
ESPHome Device Builder. Bootstrapping costs one USB flash; set
shelly_ble_proxy_ota_password in the dashboard secrets first.
* logger DEBUG with esp32_ble_client=VERBOSE / esp32_ble_tracker,
bluetooth_proxy=DEBUG, so a connect attempt logs its progress and the
failure reason.
Takes effect on the (one) USB re-flash; thereafter iterate remotely.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
openbao.k.renner.uno -> openbao.renner.uno. The k. subdomain is a
migration-era artifact, never a production convention, and is banned
per owner directive 2026-07-14 — the wiki note that claimed a
dual-hostname convention was wrong and is being corrected.
Non-public reachability is a real control, not a hostname: the public
edge serves only explicitly configured SNI (openbao is not configured),
and the new CiliumNetworkPolicy default-denies the namespace with
explicit grants (traefik->8200, ESO->8200, intra-ns for the snapshot
CronJob) matching the <app>-ns.yaml pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
The renner.uno fallthrough forwarder still pointed at the Proxmox DNS HA VIP
10.2.40.53, which is decommissioned/unreachable. Every non-Ingress renner.uno
name (device/service hostnames like udm.renner.uno) therefore dead-ended in
SERVFAIL/timeout. A routine home-assistant pod restart on 2026-07-14 forced HA
to re-resolve its device hostnames and left ~1374 entities unavailable / ~30
config entries stuck (ZHA, Shelly, UniFi, Bluetooth, cloud integrations).
Point the fallthrough at the UDM (10.2.40.1 / fd00::1), which serves the
split-horizon renner.uno zone (verified: it answers udm.renner.uno). Also
scrub the stale 10.2.40.53 references in the CoreDNS comment, bootstrap.sh
DNS_SERVER, and the Gatus monitor (which was pointlessly probing the dead VIP).
A live hotfix has already been applied to the k8s-gateway ConfigMap to restore
resolution; this makes it durable so Flux does not revert it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PwcQMntG9y9XRkNMPDq2fe
bao operator raft snapshot save nightly at 03:00 to a dedicated PVC
(keep 7), authenticated via a new k8s-auth role openbao-snapshot ->
policy raft-snapshot (read on sys/storage/raft/snapshot only; applied
2026-07-14). Velero openbao-daily (06:15) ships ONLY the snapshot PVC
offsite via label selector — the live raft data PVC stays unbacked
(crash-consistent files; the .snap is the restore artifact). A failed
Job doubles as the sealed-after-restart alarm.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
shelly-manager now auto-discovers esphome bluetooth_proxy nodes on the IoT
VLAN via mDNS (_esphomelib._tcp) — see k.renner/shelly-manager#14. Replace
the static Ble__Proxies__0__{Name,Host,Port} env (which addressed the proxy
by shelly-ble-proxy-1.iot.renner.uno — poisoned over IPv6 by the
*.renner.uno wildcard) with a single shared fleet Noise key exposed as
Ble__Discovery__Key, from the same shelly-ble-proxy-key secret.
Merge after #14's image is live so discovery is present when the static
entry goes away.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
authentik-secrets + samba-bridge config: values in OpenBao KV, both
round-trip-verified text. The identity provider was deliberately last —
every other consumer proved the pipeline first. ZERO sops files remain
under kubernetes/; authentik KS decryption dropped. Phase 4 (age-key
retirement) is now unblocked pending soak.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
seed/{authentik,immich,paperless} barman+hetzner creds and
seed/influxdb bootstrap: values in OpenBao KV (homelab/seed/*), all
round-trip-verified text. Future cold rebuilds no longer need the age
key for the seed phase. Decryption blocks dropped from every seed-* KS
(incl. dormant seed-audiobookshelf/seed-music-assistant and the
suspended, deprecated seed KS).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
home-assistant (5), immich, paperless (2): values in OpenBao KV, all
verified text-safe via base64 round-trip before load. ExternalSecrets
keep the old Secret names. Decryption blocks dropped from
home-assistant/immich/paperless KS plus the dormant one on
speech-to-phrase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
evcc, otbr, libation, external-dns (apps), monitoring (4), woodpecker:
values in OpenBao KV (homelab/apps/*, loaded 2026-07-13), manifests are
now ExternalSecrets keeping the old Secret names. libation's payload is
a binary SQLite db — stored base64 in KV with decodingStrategy: Base64
on the ES. Decryption blocks dropped from apps/evcc/otbr/libation/
monitoring KS (paths sops-free) plus the dormant one on grafana-db.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
external-dns-unifi, netbird-ingress, velero (both), cloudflare-api-token:
values live in OpenBao KV (homelab/infrastructure/*, loaded 2026-07-13),
manifests become ExternalSecrets targeting the same Secret names so
consumers see no change. Both infra KS paths are now sops-free, so their
decryption blocks go too. Rollback per file = git revert (age key still
active until phase 4).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
OpenBao was initialized and configured today (kv-v2 at homelab/, k8s
auth, eso-read policy + role bound to the external-secrets SA); the
store can therefore turn Ready and no longer endangers the post-crds
wait gate. Audit device moves into the server HCL because OpenBao 2.5
rejects API-managed audit devices; log_raw stays off so Loki only ever
sees HMAC-redacted values.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
The IoT VLAN advertises a global IPv6 prefix (RA/SLAAC), but the Atom Echo
ran v4-only because ESPHome defaults IPv6 off. Enable it so the device is a
proper dual-stack citizen of the segment and has a real global IPv6.
This is the prerequisite for the DNS-side fix: once the device has a real
AAAA, a `unifi_dns_record` AAAA for shelly-ble-proxy-1.iot.renner.uno can be
pinned to it (same pattern as the existing udm/captive records in
opentofu/dns_renner_uno.tf), so the name resolves to the device instead of
falling through to the *.renner.uno proxy wildcard.
Takes effect on the next USB re-flash (no OTA on this device by design).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Single-node raft OpenBao (5Gi ceph-block, UI at openbao.k.renner.uno,
internal-only ingress) and ESO 2.x. No ClusterSecretStore yet — it can
only turn Ready after manual init/unseal, and committing it now would
freeze the infrastructure-crds wait gate (2026-07-13 incident shape).
Full phased plan incl. per-file checklist and DR runbook in
kubernetes/MIGRATION-sops-openbao.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
shelly-manager's native ESPHome-API client (PR k.renner/shelly-manager#12)
does the real Noise_NNpsk0 handshake to the Atom Echo bluetooth_proxy when
it has the api.encryption.key. Provide it:
* shelly-ble-proxy-key.sops.yaml — SOPS Secret holding api-key, the same
value as the ESPHome dashboard's shelly_ble_proxy_api_key.
* shelly-manager-deployment: inject it as Ble__Proxies__0__Key via
secretKeyRef.
Effect: /ble-radios upgrades from a bare TCP :6053 check to a real
handshake + DeviceInfo (device name/version), which also validates the
Noise/frame stack on live hardware ahead of the BLE scan/GATT work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the LXC->in-cluster cutover the gitea scrape target became an in-cluster
pod, which the apps default-deny-ingress CiliumNetworkPolicy drops (it allows
only same-ns / home-assistant / host). Prometheus scrapes time out (context
deadline exceeded) and TargetDown keeps firing. Add a scoped CNP allowing the
monitoring namespace to reach the gitea pod on :3000. Monitoring egress is
unrestricted (its only policy is ingress-only), so this ingress allow is
sufficient.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Follow-up to the LXC retirement sweep:
- remove orphan inventory/group_vars/woodpecker_agents (group already deleted)
- drop the retired gitea-runner (root@10.2.40.214 act_runner) journalctl entry
from .vscode/settings.json
Kept (live): the ci.renner.uno gatus check and the woodpecker-ci@homelab
authorized key, which belong to the in-cluster Woodpecker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Gitea and Woodpecker now run in-cluster (k8s); their Proxmox LXCs are retired.
Remove the provisioning that rebuilt them and every reference to the dead hosts:
- delete roles gitea-lxc, gitea-runner, woodpecker-agent, woodpecker-server
- delete plays install-gitea, install-gitea-runner, configure-gitea-runner-logging,
install-woodpecker-server, install-woodpecker-agent
- drop [woodpecker_server]/[woodpecker_agents] inventory groups + group_vars
- monitoring: remove the gitea Prometheus scrape job + gitea:22 blackbox target;
repoint the gatus / blackbox-dns / chaos-test / proxmox-dns dig probes and the
dns-overview dashboard panel from gitea.srv.renner.uno to k8s-1.srv.renner.uno
- drop the retired gitea-runner journalctl entries from .vscode/settings.json
Live services keep their public hostnames (gitea.renner.uno, ci.renner.uno),
untouched. Other *.srv.renner.uno hosts untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Gitea moved in-cluster (apps ns StatefulSet); LXC 114 is stopped, so the static
`gitea` scrape target gitea.srv.renner.uno:3000 is down -> firing TargetDown.
Point it at the ClusterIP Service (gitea.apps.svc.cluster.local:3000), which
serves /metrics anonymously (verified HTTP 200 with gitea_* metrics).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Add Ble:Proxies[0] = { atom-echo-1, shelly-ble-proxy-1.iot.renner.uno, 6053 } to
the shelly-manager deployment so the flashed M5 Atom Echo esphome bluetooth_proxy
(now up at 10.2.30.109, ESPHome API :6053 open) shows on the /ble-radios page and
is liveness-probed. Reached over the same net1 macvlan route as the Shellys.
Provisioning THROUGH it is the native ESPHome-API client, still to come.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per how it's actually run: drop OTA and the fallback AP/captive_portal — this is
a fixed, low-value proxy, so updates and any WiFi recovery are done by re-flashing
over USB. Secrets shrink to iot_wifi_ssid / iot_wifi_password /
shelly_ble_proxy_api_key (the api encryption key stays; it's the ble-agent
handoff). README secrets list + flashing note updated to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#159 set SSH_USER=gitea, but SSH_USER is cosmetic (clone-URL display only) and
does not change which login the built-in SSH server accepts — so gitea@ still got
Permission denied. BUILTIN_SSH_SERVER_USER is the setting the built-in server
checks the login name against (defaults to RUN_USER=git); pin it to gitea so the
existing gitea@ remotes (the LXC's non-default SSH user) authenticate with zero
client changes. git@ stops authenticating (single built-in login) — unused here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
The retired LXC ran RUN_USER=gitea, so all clone URLs and git remotes are
gitea@gitea.renner.uno. The rootless image runs as `git`, defaulting SSH_USER to
`git` and rejecting `gitea@` logins (Permission denied). Pinning SSH_USER=gitea
restores the pre-migration convention with no per-repo remote changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Source-of-truth YAML (+ README) for the M5 Atom Echo dedicated Bluetooth proxy
that lends its BLE radio to shelly-manager's ble-agent for provisioning WiFi-less
Shellys over mOS RPC-over-BLE. Built/flashed via the in-cluster ESPHome Device
Builder (esphome.home.renner.uno).
Lean, dedicated config (no voice — bluetooth_proxy + voice boot-loops the
ESP32-PICO), esp-idf framework, bluetooth_proxy active. The api encryption key is
the handoff to the ble-agent (its BLE_PROXY_NOISE_PSK). The dashboard reads from
its PVC, not git, so these files are the reviewed/backed-up copy kept in sync by
hand.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moves ci.renner.uno off LXC 231 onto the in-cluster StatefulSet (DB loaded from
the LXC beforehand) and brings the kubernetes-backend agent online, retiring the
docker agents (LXC 910/911). server replicas 0->1; Service ExternalName->ClusterIP
with http:8000 (UI) + grpc:9000 (agent<->server); agent replicas 0->1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Moves gitea.renner.uno and git-over-ssh onto the in-cluster StatefulSet, which
has been loaded from LXC 114 and verified (data intact, SSH host-key identity
preserved, master at the revision Flux expects). gitea Service ExternalName ->
ClusterIP; adds the gitea-ssh IngressRouteTCP binding the Traefik entrypoint
(added in phase 1) to :2222. Woodpecker stays on the LXC (server replicas 0,
Service ExternalName) — its cutover is a separate migration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
woodpecker-server envFrom's woodpecker-secret; without it the pod crashloops and
holds the apps Kustomization (wait:true) red. Encrypted with the repo age
recipient (kubernetes/.sops.yaml); Flux decrypts it via the apps Kustomization's
sops provider. Values (AGENT_SECRET/GITEA_CLIENT/GITEA_SECRET) lifted from the
running CT 231 woodpecker-server.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Stand up Gitea + Woodpecker as in-cluster workloads beside the existing
Proxmox LXCs WITHOUT flipping any traffic. Both Services remain ExternalName
-> the LXCs, the Woodpecker agent ships replicas:0, and the gitea SSH
IngressRouteTCP is absent. Merging this changes nothing user-facing — Flux
brings up empty-PVC pods that get loaded and verified out-of-band before a
phase-2 PR flips the Services to ClusterIP.
Gitea: gitea/gitea:1.21.4-rootless (its entrypoint runs environment-to-ini,
so do NOT override it), built-in SSH server on 2222, forced GITEA__log__ROOT_PATH,
host-key reuse. Traefik gains an additive gitea-ssh TCP entrypoint (:22 on the
shared VIP). Woodpecker: kubernetes backend (build pods in ns woodpecker-ci on
ceph-filesystem RWX, privileged-plugin allow-list + PodSecurity label), agent
replicas:0; SOPS secret is created+committed at pre-flight, never kubectl-applied.
Cutover is fully GitOps: flux suspend + kubectl exec/scale data load, verify
via port-forward, phase-2 flip with a repo-resync; break-glass = repoint the
Flux source to the GitHub mirror (known-good tag before-k8s-migration). See
kubernetes/apps/MIGRATION-gitea-woodpecker.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
2026-07-12 20:55:15 +02:00
554 changed files with 17892 additions and 13660 deletions
kubectl get --raw "$$PROM/api/v1/query?query=max%20by%20(alertname,namespace,job,severity)%20(max_over_time(ALERTS%7Balertstate%3D%22firing%22,alertname!~%22Watchdog%7CInfoInhibitor%22%7D%5B35m%5D))" \
annotations: {summary: "BLIP: fired within the lookback window (~35 min) and already resolved — caught by the ALERTS history scan, not currently active."}}' \
# Headless triage run. Same auth model as security-scan.yml
# (subscription OAuth, prompt piped via stdin — see the comments
# there). Tool allowlist is read-only on the repo plus Write for the
# /tmp verdict file, plus EXACTLY ONE live command: the RBAC-scoped
# kubectl from make-diag-kubeconfig.sh (read-only ClusterRole, no
# Secrets). The prefix match means no pipes/redirects and no other
# binaries — the agent verifies cluster state, it cannot mutate it,
# fetch the web, or read secrets.
export KUBECONFIG="$$PWD/.ci-diag/kubeconfig"
# Model chosen live from the catalog (not hardcoded, no family assumed) —
# see the header of scripts/ci/pick-model.sh. Empty MODEL -> CLI default.
MODEL=$$(sh scripts/ci/pick-model.sh "triage firing Prometheus/Alertmanager alerts against read-only live cluster state and classify each as a real problem or noise; careful diagnostic reasoning, no changes")
if [ -n "$$MODEL" ]; then echo "alert-monitor: selected model $$MODEL"; else echo "alert-monitor: no live catalog — using the CLI default model"; fi
cat ci/agents/alert-monitor/prompt.md | claude $${MODEL:+--model $$MODEL} \
echo "Issue already open for '$$TITLE' — skipping."
else
jq --arg footer "$$(printf '\n\n---\n<!-- alert-monitor:fingerprint=%s -->\n_Filed automatically by the alert-monitor pipeline (%s). The issue-fix pipeline may pick this up and open a PR._' "$$FP" "$$STAMP")" \
printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state." > /tmp/noise-header.txt
# tools plus EXACTLY ONE live command: the RBAC-scoped read-only
# kubectl from fetch-diag (no Secrets readable, prefix match forbids
# pipes and other binaries). No web (see security-scan.yml).
[ -f /tmp/alert-fix/issue.json ] || exit 0
export KUBECONFIG="$$PWD/.ci-diag/kubeconfig"
# Model chosen live from the catalog (not hardcoded, no family assumed) —
# see the header of scripts/ci/pick-model.sh. Empty MODEL -> CLI default.
MODEL=$$(sh scripts/ci/pick-model.sh "diagnose an [alert] issue against read-only live cluster state and fix the root cause by editing repo manifests; multi-step reasoning that changes infrastructure code and needs a capable model")
if [ -n "$$MODEL" ]; then echo "issue-fix: selected model $$MODEL"; else echo "issue-fix: no live catalog — using the CLI default model"; fi
cat ci/agents/issue-fix/prompt.md | claude $${MODEL:+--model $$MODEL} \
if git diff --quiet && git diff --cached --quiet \
&& [ -z "$$(git ls-files --others --exclude-standard)" ]; then
echo "Agent made no edits — commenting assessment on issue #$$NUM."
[ -s /tmp/alert-fix/summary.md ] \
|| echo "Agent made no edits and wrote no summary." > /tmp/alert-fix/summary.md
printf '\n\n<!-- alert-fix:assessed -->\n_Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._\n' \
>> /tmp/alert-fix/summary.md
jq -n --rawfile body /tmp/alert-fix/summary.md '{body: $$body}' > /tmp/gitea-comment.json
# latter passes a multi-KB substituted string into the command
# line which busybox ash's tracer (set -x) misparsed under
# woodpecker, producing a spurious "unterminated quoted string".
cat ci/agents/security-fix/prompt.md | claude \
# Model chosen live from the catalog (not hardcoded, no family assumed) —
# see the header of scripts/ci/pick-model.sh. Empty MODEL -> CLI default.
MODEL=$$(sh scripts/ci/pick-model.sh "fix CRITICAL/HIGH CVEs and IaC misconfigurations (Trivy findings) in Kubernetes/Ansible manifests by editing files; multi-step, security-sensitive work touching container versions, RBAC and configuration that needs careful, capable reasoning")
if [ -n "$$MODEL" ]; then echo "security-fix: selected model $$MODEL"; else echo "security-fix: no live catalog — using the CLI default model"; fi
cat ci/agents/security-fix/prompt.md | claude $${MODEL:+--model $$MODEL} \
# security_bot_token is scoped for code/PRs only — its issue POST
# returned HTTP 403 "token does not have ... write:issue", so the
# audit ran green but never created an issue. Switch to gitea_token,
# the broader machine account already used by build-caddy /
# build-ci-images for release + registry writes, which is the most
# likely existing secret to also carry write:issue. Audit issues are
# then attributed to this token's owner rather than the security bot.
# Dedicated identity: update-audit-bot, PAT scoped to write:issue ONLY.
#
# This resolves the gap #197 tracked. The history: security_bot_token is
# scoped for code/PRs, so its issue POST returned 403 and the audit ran
# green while creating nothing; the fix at the time was the admin-owned
# gitea_token, which handed a broad credential to a WebFetch-capable
# agent that reads untrusted third-party release pages. That is now a
# narrow token that cannot do anything but write issues — verified 403
# on /user/repos and /admin/users. See #330.
#
# The other mitigations still apply and are still load-bearing: tool
# scoping (no WebSearch), the Read sandbox in settings.json, and the
# prompt hardening that treats fetched content as untrusted data.
GITEA_TOKEN:
from_secret:gitea_token
from_secret:update_audit_bot_token
commands:
- |
# Hand off to Claude Code in headless mode. Same auth model as the
@@ -51,9 +57,38 @@ steps:
# pins, fetches upstream releases, and writes a report to /tmp. It
# has NO Edit/Bash, so it cannot mutate the repo. Prompt is piped via
# stdin (see security-scan.yml for why -p "$(cat ...)" is avoided).
cat ci/agents/update-audit/prompt.md | claude \
#
# Security hardening (gitea issue #197 — prompt-injection exfil via
# the untrusted release pages this agent fetches):
# - WebSearch is dropped. It can't be domain-scoped (unlike
# WebFetch) and the prompt already gives explicit source URLs or
# a well-known registry (Artifact Hub / chart source repo) for
# everything it needs to look up.
# - --settings loads ci/agents/update-audit/settings.json, which
# denies Read on /proc, /root, /home, /etc, and this repo's own
# .git/ — closing off the credential-file and env-dump read
# paths a prompt-injected instruction could otherwise try.
# (Write path-scoping was tested and found NOT to be enforced
# under --permission-mode acceptEdits in the current Claude Code
# CLI, so it's intentionally not relied on here; the pipeline
# only ever trusts /tmp/update-audit.md regardless of what else
# the agent writes.)
# - prompt.md now tells the agent fetched content is untrusted
# data, not instructions, and to never echo it verbatim.
# A full WebFetch domain allowlist was considered and rejected: the
# Helm-chart-version check legitimately needs to reach whichever
# chart-repo domain each HelmRelease's own (already-trusted)
# HelmRepository/OCIRepository URL points at — a fixed allowlist
# across ~20 charts would need updating every time a chart is added
# and would silently stop auditing anything not yet listed.
# Model chosen live from the catalog (not hardcoded, no family assumed) —
# see the header of scripts/ci/pick-model.sh. Empty MODEL -> CLI default.
MODEL=$$(sh scripts/ci/pick-model.sh "weekly, read-only homelab infrastructure update audit: compare Talos/Kubernetes/Flux/Helm/Ansible/OpenTofu version pins against upstream releases and write a drift report; moderate-difficulty analysis, no code changes")
if [ -n "$$MODEL" ]; then echo "update-audit: selected model $$MODEL"; else echo "update-audit: no live catalog — using the CLI default model"; fi
cat ci/agents/update-audit/prompt.md | claude $${MODEL:+--model $$MODEL} \
# Safety switch: must be set to true explicitly. This play is archived
# (one-off cleanup of an accidental install) and deletes the gitea
# service/binary/config/user on every host except gitea.srv.renner.uno
# if run unguarded.
confirm_destroy:false
tasks:
- name:Safety Check
ansible.builtin.fail:
msg:"This archived playbook removes the gitea service/binary/config/user on all hosts except gitea.srv.renner.uno. You must run with -e 'confirm_destroy=true'."
{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"histogram_quantile(0.50, sum(rate(authentik_main_request_duration_seconds_bucket[5m])) by (le))","legendFormat":"p50","refId":"A"},
{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"histogram_quantile(0.95, sum(rate(authentik_main_request_duration_seconds_bucket[5m])) by (le))","legendFormat":"p95","refId":"B"},
{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"histogram_quantile(0.99, sum(rate(authentik_main_request_duration_seconds_bucket[5m])) by (le))","legendFormat":"p99","refId":"C"}
{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"histogram_quantile(0.95, sum(rate(authentik_flows_execution_stage_time_bucket[5m])) by (le, stage_type))","legendFormat":"{{stage_type}}","refId":"A"}
{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"histogram_quantile(0.95, sum(rate(authentik_flows_plan_time_bucket[5m])) by (le))","legendFormat":"Flow Plan p95","refId":"A"},
{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"histogram_quantile(0.95, sum(rate(authentik_flows_stage_time_bucket[5m])) by (le))","legendFormat":"Stage Time p95","refId":"B"}
"targets":[{"expr":"topk(15, sum by (host, id) (label_replace(rate(container_cpu_usage_seconds_total{id=~\".*/docker-[a-f0-9]+\\\\.scope\"}[5m]), \"container\", \"$1\", \"id\", \".*/docker-([a-f0-9]{12}).*\")))","legendFormat":"{{host}}/{{container}}","refId":"A"}],
"title":"CPU Usage by Container (Top 15, Stacked)",
"targets":[{"expr":"topk(10, sum by (host, id) (label_replace(rate(container_network_receive_bytes_total{id=~\".*/docker-[a-f0-9]+\\\\.scope\"}[5m]), \"container\", \"$1\", \"id\", \".*/docker-([a-f0-9]{12}).*\")))","legendFormat":"{{host}}/{{container}}","refId":"A"}],
"targets":[{"expr":"topk(10, sum by (host, id) (label_replace(rate(container_network_transmit_bytes_total{id=~\".*/docker-[a-f0-9]+\\\\.scope\"}[5m]), \"container\", \"$1\", \"id\", \".*/docker-([a-f0-9]{12}).*\")))","legendFormat":"{{host}}/{{container}}","refId":"A"}],
"targets":[{"expr":"topk(10, sum by (host, id) (label_replace(rate(container_fs_reads_bytes_total{id=~\".*/docker-[a-f0-9]+\\\\.scope\"}[5m]), \"container\", \"$1\", \"id\", \".*/docker-([a-f0-9]{12}).*\")))","legendFormat":"{{host}}/{{container}}","refId":"A"}],
"targets":[{"expr":"topk(10, sum by (host, id) (label_replace(rate(container_fs_writes_bytes_total{id=~\".*/docker-[a-f0-9]+\\\\.scope\"}[5m]), \"container\", \"$1\", \"id\", \".*/docker-([a-f0-9]{12}).*\")))","legendFormat":"{{host}}/{{container}}","refId":"A"}],
{"asDropdown":false,"icon":"external link","includeVars":false,"keepTime":true,"tags":[],"targetBlank":false,"title":"Back to Overview","type":"link","url":"/d/home-overview/home-overview"}
{"asDropdown":false,"icon":"external link","includeVars":false,"keepTime":true,"tags":[],"targetBlank":false,"title":"Back to Overview","type":"link","url":"/d/home-overview/home-overview"}
"targets":[{"expr":"homeassistant_sensor_power_w{friendly_name=\"Smart Meter Sum active instantaneous power\"}","legendFormat":"Total Power","refId":"A"}],
{"expr":"homeassistant_sensor_power_w{friendly_name=~\"Smart Meter L[123] active instantaneous power\"}","legendFormat":"{{friendly_name}}","refId":"A"}
"targets":[{"expr":"homeassistant_sensor_power_w{friendly_name=\"Smart Meter Sum active instantaneous power\"}","legendFormat":"Total Power","refId":"A"}],
"targets":[{"expr":"homeassistant_sensor_voltage_v{friendly_name=~\"Smart Meter L[123] active instantaneous voltage\"}","legendFormat":"{{friendly_name}}","refId":"A"}],
"targets":[{"expr":"count(homeassistant_sensor_battery_percent < 20 and homeassistant_sensor_battery_percent > 0) or vector(0)","legendFormat":"Low Batteries","refId":"A"}],
{"asDropdown":false,"icon":"external link","includeVars":false,"keepTime":true,"tags":[],"targetBlank":false,"title":"Back to Overview","type":"link","url":"/d/home-overview/home-overview"}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
issue-fix-bot
security-bot
gitea_admin