Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5 c133f44d28 docs(zigbee2mqtt): correct the network_key extraction comment
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review point on #444, which merged before this could be folded in.

The comment claimed the listed shapes yield "null" at rc=0. That describes
BARE `.network_key.key`, not the expression actually written, and it
contradicted its own closing line two lines below. Measured against jq
1.7.1 in the image, `(.network_key?.key? // "")` returns an EMPTY string at
rc=0 for all of:

  {}  {"devices":[]}  {"network_key":{}}  {"network_key":null}
  {"network_key":"abc"}  {"network_key":123}
  []  "a string"

and never the literal text null, because `// ""` maps a null result to
empty. Bare `.network_key.key` is what exits 5 on the last four shapes.

Checking that also turned up something the review did not mention: because
KEY can never be "null" with the current expression, the `[ "$KEY" =
"null" ]` arm is dead code today. Kept deliberately as belt-and-braces for
a future edit that drops the `// ""` fallback, and now labelled as such
rather than left looking like the arm that fires.

Comment-only: `git diff` shows zero changed lines outside comments, and the
rendered script is unchanged in behaviour.

NOTE: this text lives in the Deployment's init container command, so it is
part of the pod template — merging rolls the zigbee2mqtt pod (~20s of
Zigbee downtime, replicas: 1 + Recreate). Incidental benefit: the restart
resets Z2M's in-memory configure attempt counter, giving the Philips
devices that are interviewed but not yet bound (both RDM001 Treppenhaus
switches, the RWL022, and Sarah) three fresh configure attempts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 12:30:23 +02:00
k.renner 4c994d1ce3 Merge pull request 'refactor(zigbee2mqtt): use alpine-jq image, drop apk add from seed-data' (#444) from rename-alpine-jq into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#444
2026-07-30 10:19:19 +00:00
claude-botandClaude Opus 5 8439a99a98 fix(zigbee2mqtt): guard the network_key selector at both levels
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Round-3 review on #444. Both suggestions were optional; both were cheap to
close properly.

`.network_key.key?` only shielded the leaf selector. If the backup's TOP
level were not an object, `.network_key` itself raises before `?` applies.
Verified in the image with jq 1.7.1:

  []          (.network_key.key? // "")   -> rc=5 Cannot index array ...
  "a string"  (.network_key.key? // "")   -> rc=5 Cannot index string ...
  []          (.network_key?.key? // "")  -> rc=0 empty
  "a string"  (.network_key?.key? // "")  -> rc=0 empty

So `?` now sits on both selectors. Not a realistic shape for an Open
Coordinator Backup export and it failed closed either way, but it was the
last path that produced jq's raw error rather than a tailored FATAL line.

Full matrix against the rendered script, on the alpine-jq:2026-07-30 tag:

  real backup            -> exit 0, hash 27c5c77a945bcecc (matches live)
  non-hex key            -> FATAL: contains non-hex characters
  4-char key             -> FATAL: length 4, expected 32 hex chars
  no network_key         -> FATAL: .network_key.key absent
  network_key non-object -> FATAL: .network_key.key absent
  network_key number     -> FATAL: .network_key.key absent
  top-level array        -> FATAL: .network_key.key absent
  top-level string       -> FATAL: .network_key.key absent
  device link_key in doc -> correctly ignored; network_key used

On the registry suggestion: re-verified the tag pulls immediately before
merge — jq 1.7.1, alpine 3.21.7. It was published by pipeline 3060, the
push build for this branch. No build ran for the later commits because they
touch only kubernetes/apps/**, which correctly does not match the build
step's when.path; the image content is unchanged by them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 10:45:38 +02:00
claude-botandClaude Opus 5 6e4729c7d0 fix(zigbee2mqtt): harden network_key extraction against malformed shapes
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Round-2 review on #444.

The 🟡 identified a real gap, though not via the input shape it described.
Verified against jq 1.7.1 in the actual image:

  {"devices":[]}            -> "null", rc=0   (absent path is LENIENT)
  {"network_key":{}}        -> "null", rc=0
  {"network_key":null}      -> "null", rc=0
  {"network_key":"abc"}     -> rc=5, Cannot index string with string "key"
  {"network_key":123}       -> rc=5, Cannot index number with string "key"

So a missing network_key was already handled correctly and did produce the
intended FATAL line. What was NOT handled is a network_key that exists but
is not an object: plain `.key` exits 5 there, and under `set -e` that
aborts before the checks run, surfacing jq's raw error instead. Fixed with
`.network_key.key?`, which yields empty for those shapes and routes them
into the existing absent-key branch.

Now every malformed shape reports the same readable message:

  real backup            -> exit 0, hash 27c5c77a945bcecc (matches live)
  non-hex key            -> FATAL: contains non-hex characters
  4-char key             -> FATAL: length 4, expected 32 hex chars
  no network_key         -> FATAL: .network_key.key absent
  network_key non-object -> FATAL: .network_key.key absent
  network_key number     -> FATAL: .network_key.key absent
  device link_key in doc -> correctly ignored; network_key used

Verification ran against the alpine-jq:2026-07-30 tag itself.

On the 🔴: the premise does not hold — that tag already exists and is
pullable, verified twice by pulling it. The build step fires on branch
pushes as well as master, so pipeline 3060 published it when this branch
was pushed; the reviewer could not query the registry to see that. The
underlying caution is still worth recording, so the manifest now documents
that TAG is derived from `date` AT RUN TIME, that a date must never be
written here before a build has produced it, and that a later master build
publishes its own tag without deleting this one — so this keeps resolving,
though the image then came from a branch build rather than master. Also
recorded why it matters: Flux reconciles on a 10m poll with no ordering
against kaniko, and replicas: 1 + Recreate means an unpullable init image
would leave the sole coordinator in ImagePullBackOff.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 10:39:57 +02:00
claude-botandClaude Opus 5 f66b127035 refactor(zigbee2mqtt): pin alpine-jq to a dated tag, not :latest
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Review point on #444. The reviewer cited a precedent I had missed:
ci-ansible:2026-04-29 in kubernetes/infrastructure/netbird-backup/
cronjob.yaml. Verified — for self-built images referenced by tag rather
than digest, this repo already uses DATED tags, with :latest used nowhere.

The argument holds independently of convention: with :latest, rebuilding
this base for some unrelated consumer silently changes what the seed-data
init container runs on its next pod restart, with no diff here to review.
That container renders the Zigbee network key, so "what exactly does it
run" should be visible in the manifest. A dated tag makes a change to it a
deliberate, reviewable act.

This does give up automatic pickup of rebuilds, which is the tradeoff the
previous commit chose the other way round. Given the base is thin and
rarely changes, and the consumer is security-sensitive, reviewability wins
— and it stays consistent with the owner directive that our own images do
not need digest pinning, since a dated tag is not a digest.

Verified gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30 pulls and
carries jq 1.7.1, so the reference resolves before merge rather than after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 10:01:13 +02:00
claude-botandClaude Opus 5 c3280310b5 refactor(zigbee2mqtt): use alpine-jq image, drop apk add from seed-data
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Owner directives, both applied here:

- "z2m-seed is a shit name for the image — name it for what it is, not for
  how it is used." Renamed to alpine-jq. Nothing in it is
  zigbee-specific; it is alpine plus jq, useful to anything that needs to
  read JSON from a shell script. The Containerfile now describes it that
  way, with the zigbee2mqtt case recorded as its first consumer rather
  than its identity.
- "You do not need to pin the digest for our own images." Referenced by
  tag. This also collapses what was going to be two PRs into one, since
  the digest no longer has to exist before the Deployment can name it.

The seed-data init container now uses that image instead of
`apk add --no-cache jq` on every pod start.

Two consequences of dropping digest pinning, both good:

- The whole --image-name-with-digest-file / "PIN THIS:" apparatus is gone
  from the pipeline. That machinery existed only to hand a digest to a
  follow-up commit, and it is what broke every pipeline in the repo (#443)
  via an unquoted ": " in a YAML scalar. Less surface, same outcome.
- :latest means Kubernetes defaults imagePullPolicy to Always, so a
  rebuilt image is picked up without editing the manifest.

Kept the fail-loudly validation, since a wrong network_key costs the whole
Zigbee network: explicit checks for an absent key (jq prints "null" for a
missing path, so that is checked separately rather than being reported as
merely "non-hex"), non-hex characters, and a length other than 32.

Verified by running the manifest's RENDERED script, against the real
coordinator backup and synthetic fixtures. The alpine-jq tag does not
exist until this merges and builds, so the run used the existing image of
identical content (alpine 3.21.7 + jq 1.7.1, verified present):

  real backup          -> exit 0, network_key line byte-identical to the
                          live value (sha256 27c5c77a945bcecc)
  non-hex key          -> exit 1, "contains non-hex characters"
  4-char key           -> exit 1, "length 4, expected 32 hex chars"
  absent network_key   -> exit 1, ".network_key.key absent from ..."
  device link_key also
  present in the doc   -> correctly ignored; the network_key is used

Validated with `woodpecker lint` (not just `yq`, which is what let #443
through), plus kustomize build and a server-side dry run. No stale
z2m-seed references remain.

NOTE: the now-orphaned z2m-seed package is still in the registry and
should be deleted; that needs package-delete rights this account does not
have.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 09:55:03 +02:00
k.renner d1b3521b7f Merge pull request 'fix(ci): repair pipeline config broken by unquoted ": " in echo' (#443) from fix-ci-yaml into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#443
2026-07-30 07:31:57 +00:00
4 changed files with 101 additions and 52 deletions
+10 -28
View File
@@ -131,19 +131,19 @@ steps:
--destination gitea.renner.uno/gitea_admin/secret-rotator:$TAG \
--destination gitea.renner.uno/gitea_admin/secret-rotator:latest
# Runtime image, not CI tooling: alpine + jq, the base for the zigbee2mqtt
# `seed-data` init container. Replaces an `apk add --no-cache jq` that ran on
# every pod start — tools belong in the image, not installed at runtime.
# Consumers pin it by DIGEST (as the secret-rotator CronJobs do), so this must
# be built and pushed before the Deployment can reference it.
- name: build-z2m-seed
# Runtime image, not CI tooling: a general-purpose alpine + jq base for
# workloads that need to read JSON from a shell script. Named for what it IS,
# not for who uses it — its first consumer is the zigbee2mqtt seed-data init
# container, but nothing about it is zigbee-specific. Exists so tools come from
# the image rather than an `apk add` on every container start.
- name: build-alpine-jq
image: gcr.io/kaniko-project/executor:v1.23.2-debug
directory: ci/images/z2m-seed
directory: ci/images/alpine-jq
depends_on: []
when:
- event: manual
- event: push
path: ["ci/images/z2m-seed/**", ".woodpecker/build-ci-images.yml"]
path: ["ci/images/alpine-jq/**", ".woodpecker/build-ci-images.yml"]
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
@@ -157,23 +157,5 @@ steps:
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/z2m-seed:$TAG \
--destination gitea.renner.uno/gitea_admin/z2m-seed:latest \
--image-name-with-digest-file /kaniko/z2m-seed.digest
# Print the digest-pinned reference so the follow-up commit that pins the
# Deployment can copy it straight from this build log, with no separate
# registry query.
#
# kaniko writes ONE LINE PER --destination to that file, so it holds two
# entries (:$TAG and :latest) carrying the same digest. Reduce to the bare
# digest and dedupe, so exactly one unambiguous line is emitted rather than
# two near-identical ones for whoever does step 2 to choose between.
- DIGEST=$(cut -d@ -f2 /kaniko/z2m-seed.digest | sort -u | head -1)
# MUST stay single-quoted as one YAML scalar. Unquoted, the ": " inside
# makes YAML parse the whole command as a MAP rather than a string, and the
# pipeline fails to compile with
# cannot unmarshal 'map[echo "PIN THIS:...' into a string value
# which errors EVERY pipeline in the repo, not just this one. Note `yq`
# accepts the unquoted form because it is valid YAML — just a map, not a
# string — so validate with `woodpecker lint`, not a YAML parser.
- 'echo "PIN THIS: gitea.renner.uno/gitea_admin/z2m-seed@$DIGEST"'
--destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
--destination gitea.renner.uno/gitea_admin/alpine-jq:latest
+24
View File
@@ -0,0 +1,24 @@
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
# Alpine plus jq. Nothing application-specific — a general-purpose base for any
# workload that needs to read JSON from a shell script.
#
# It exists because installing tools at container start is not acceptable here:
# `apk add` makes every start depend on network and package-index availability,
# and the resolved version can drift between restarts of an otherwise pinned
# image. Tools belong in the image.
#
# Everything else such a script typically needs (sh, sed, cp, cat, tr, awk) comes
# from busybox in the base, so jq is the only addition.
#
# Runs as root like the other ci/images entries. Consumers that can drop
# privilege should set runAsNonRoot/runAsUser themselves — what the workload
# needs to read is workload-specific, so it does not belong baked in here.
#
# First consumer is the zigbee2mqtt seed-data init container. Worth recording why
# a real JSON parser is justified there rather than a sed/awk match: "key" occurs
# 12 times in that document (once per device with an APS link_key) and the FIRST
# occurrence is a DEVICE key, because "devices" sorts before "network_key". An
# unanchored extraction therefore yields a device link key, and herdsman then
# forms a NEW Zigbee network and drops every device.
RUN apk add --no-cache jq
-21
View File
@@ -1,21 +0,0 @@
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
# Runtime image, not CI tooling: the base for the zigbee2mqtt `seed-data` init
# container.
#
# It exists for exactly one reason — jq. That init container reads
# `.network_key.key` out of the Open Coordinator Backup JSON to render
# /app/data/secret.yaml; everything else it does (sh, sed, cp, cat) is already
# in busybox. jq is baked in here rather than `apk add`ed when the container
# starts, so the init container has no network dependency at start and no
# package-index drift between pod restarts.
#
# Parsing this with busybox sed/awk instead was considered and rejected: "key"
# occurs 12 times in the real backup (once per device with an APS link_key) and
# the FIRST occurrence is a device key, because "devices" sorts before
# "network_key". Hand-rolled extraction that isn't correctly anchored yields a
# device link key, and herdsman then forms a NEW network and drops every
# device — so a real JSON parser earns its keep here.
#
# Stays root: the init container writes to the root-owned zigbee2mqtt-data PVC.
RUN apk add --no-cache jq
@@ -39,16 +39,80 @@ spec:
# network_key, the latter extracted from the OpenBao coordinator backup
# as a natural-byte-order int array.
- name: seed-data
image: alpine:3.21
# alpine + jq, built by .woodpecker/build-ci-images.yml from
# ci/images/alpine-jq/. Replaces `apk add --no-cache jq` on every pod
# start: tools belong in the image, not installed at runtime.
# Referenced by tag rather than digest — these are our own images.
# DATED tag, not :latest, matching the ci-ansible:2026-04-29 precedent
# in kubernetes/infrastructure/netbird-backup/cronjob.yaml. With
# :latest, rebuilding this base for some unrelated consumer would
# silently change what this init container runs on its next pod
# restart, with no diff here to review — and this is the container that
# renders the Zigbee network key. Bumping the tag is a deliberate,
# reviewable act; the cost is remembering to bump it.
#
# This tag ALREADY EXISTS and was verified pullable before merge — the
# build step also fires on branch pushes, so it was published when this
# branch was pushed. That matters because Flux reconciles this directory
# on a 10m poll with no ordering against the kaniko pipeline, and with
# replicas: 1 + strategy: Recreate an unpullable init image would leave
# the sole Zigbee coordinator in ImagePullBackOff.
#
# Caveat when bumping: the build derives TAG from `date` AT RUN TIME, so
# never write a date here that a build has not already produced. If a
# later master build runs on a different day it publishes that day's tag
# and leaves this one untouched (tags are not deleted), so this keeps
# resolving — but the image then came from the branch build rather than
# master. Re-point it at a master-built tag when convenient.
image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
command:
- sh
- -c
- |
set -eu
umask 077
apk add --no-cache jq >/dev/null
# Read the network key with a real JSON parser. NOT a hand-rolled
# sed/awk match: "key" occurs 12 times in the real backup (once per
# device with an APS link_key) and the FIRST occurrence is a DEVICE
# key, because "devices" sorts before "network_key". An unanchored
# extraction yields a device link key, and herdsman then forms a
# NEW network and drops every device.
# `?` on BOTH selectors, plus the `// ""` fallback, so every
# malformed shape reaches the checks below with a readable message
# instead of aborting on jq's raw error under `set -e`.
#
# Measured against jq 1.7.1 in this image. The expression AS WRITTEN
# returns an EMPTY string at rc=0 for every one of:
# {} {"devices":[]} {"network_key":{}} {"network_key":null}
# {"network_key":"abc"} {"network_key":123} (parent not an object)
# [] "a string" (top level not an object)
# and the key itself for a well-formed backup.
#
# The guards are what make that true. Bare `.network_key.key` exits
# 5 with `Cannot index ... with string "key"` on the last four
# shapes, and `set -e` would abort right here, surfacing jq's error
# instead of the FATAL lines below.
KEY=$(jq -r '(.network_key?.key? // "")' /backup/coordinator_backup.json)
# Fail LOUDLY rather than render a wrong key, for the same reason.
# `-z` is the arm that actually fires today: `// ""` maps a null
# result to empty, so KEY is never the literal text "null" with the
# expression above. The "null" arm is belt-and-braces for a future
# edit that drops the fallback, where jq WOULD print `null`. Both
# mean "absent" — reporting them as merely "non-hex" would send you
# looking at the key instead of at the missing field.
if [ "$KEY" = "null" ] || [ -z "$KEY" ]; then
echo "FATAL: .network_key.key absent from coordinator_backup.json" >&2
exit 1
fi
case "$KEY" in
*[!0-9a-fA-F]*) echo "FATAL: network_key contains non-hex characters" >&2; exit 1 ;;
esac
if [ "${#KEY}" -ne 32 ]; then
echo "FATAL: network_key length ${#KEY}, expected 32 hex chars" >&2
exit 1
fi
# hex key "fe0b..2d" -> "0xfe,0x0b,..,0x2d" (natural order)
NK=$(jq -r '.network_key.key' /backup/coordinator_backup.json | sed 's/../0x&,/g; s/,$//')
NK=$(printf '%s' "$KEY" | sed 's/../0x&,/g; s/,$//')
cat > /app/data/secret.yaml <<EOF
mqtt_password: "${MQTT_PASSWORD}"
network_key: [${NK}]