ETP=Local broke cross-VLAN reach because workstations on VLAN 50 traverse
the UDM gateway L3, which lands traffic on whichever node it has ARP for —
with Local that node may not host the backend pod, producing TCP RST and
silent UDP drop. Match the default services-pool house pattern (ingress-nginx,
kube-dns-lb) which both run ETP=Cluster.
Drop the source_ip label promotion since under ETP=Cluster it would carry
a cluster node IP rather than the sender's IP.
Add an ingress-nginx Ingress for the in-cluster Loki at loki.k.renner.uno
gated by HTTP basic auth (htpasswd secret in sops) and TLS via
letsencrypt-prod, so the workstation can point logcli at the K8s Loki via
LOKI_ADDR/LOKI_USERNAME/LOKI_PASSWORD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HelmRelease pointed at https://k8snetworkplumbingwg.github.io/helm-charts
which 404s — the upstream helm-charts repo only ships charts for
local-clone install, not as a published Helm repository. Fetched in
fallback by Flux's source-controller failed and infrastructure stalled
in Reconciling, blocking everything that dependsOn it (matter-server
included).
Vendor the upstream daemonset manifest pinned to v4.2.4. Going thick
because v4.2.4-thin and unsuffixed v4.2.4 are not published as image
tags — only v4.2.4-thick / v4.2.4-debug exist for the release. The
thick variant chroots into /hostroot to exec CNI plugins; Talos's
/opt/cni/bin is writable (Cilium already uses it) so this is fine.
Header comment captures the why-vendored / why-thick / Talos-OK
reasoning so the next reader doesn't have to re-derive it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The prior agent's hostNetwork: true was the wrong abstraction. Earlier
in this PR I tried a Cilium-L2 LoadBalancer Service on 10.2.30.226,
but that only forwards a single TCP port (5580 ws) and does not give
the pod L2 presence on the IoT subnet — outbound is still cluster pod
network, no multicast, no Matter operational ports. Matter device
commissioning and discovery need real mDNS send/receive on the IoT L2.
Install multus-cni alongside Cilium and attach matter-server to a
NetworkAttachmentDefinition that does macvlan on k8s-edge's eth1 (the
v30iot bridge — verified via `talosctl -n k8s-edge get routes`,
10.2.30.0/24 is connected on eth1). Static IPAM pins 10.2.30.226 with
no gateway, so default-route traffic stays on the Cilium-managed
primary interface and only IoT-subnet traffic crosses net1.
The wiki note hostnetwork-app-workload-antipattern still applies, but
unlike the otbr precedent (commit 1972c9b) — where ClusterIP alone is
sufficient — matter-server actually does need IoT-VLAN reach, just not
via hostNetwork. multus is the documented escape hatch the wiki has
flagged but never exercised in this homelab; first install lands here.
iot-pool ledger now: .224 = home-assistant (LB), .225 = mosquitto
(LB), .226 = matter-server (multus, *not* a Cilium LB). LB Services
all pin explicit IPs via lbipam.cilium.io/ips so no auto-allocation
collides with the multus IP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stand up a dedicated alloy-syslog Deployment in monitoring that listens
on UDP/514, UDP/1514 (legacy parity), TCP/514 and TCP/6514+TLS and
writes into the in-cluster Loki SingleBinary.
Exposed as a Cilium L2 LoadBalancer pinned to 10.2.40.84 and published
as syslog.k.renner.uno via external-dns; that meant adding `service` to
external-dns-unifi sources (was ingress-only). TLS cert issued by
letsencrypt-prod through cert-manager.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OTBR's only outbound need is the spinel TCP connection to
slzb-thread.renner.uno:6638. HA and matter-server reach OTBR's REST
API via the ClusterIP Service. None of that requires the host's
network namespace. The wiki's hostnetwork-app-workload-antipattern
applies here.
If border-router coordination on the IoT VLAN ever becomes a real
requirement, the right answer is a Cilium-L2 LoadBalancer pinned IP
(same shape as home-assistant-iot / mosquitto-iot), not hostNetwork.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SLZB exposes the Thread RCP via Serial-over-IP at port 6638 per the
smlight docs. I had 9999 from a placeholder I made up in a comment
yesterday and never verified — port 9999 is closed on the device.
6638 is open (`nc -z` confirmed before commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After NAT64=0, the next entrypoint stage that fails is firewall_start
(otbr-firewall service) which runs ip6tables -t filter ... — that
table also doesn't exist on Talos. The firewall sets up
mesh→outside ingress filtering; routing itself doesn't need it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The otbr image's nat64_start → nat44_start init does
`iptables -t nat/mangle/filter ...` and Talos doesn't load those
kernel modules by default → "table does not exist" → entrypoint
exits 1. HA only consumes OTBR's REST API on :8081, not the
Thread→IPv4 NAT path, so disable both.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace TBD-SLZB-HOSTNAME placeholder. Note: yesterday's
investigation found OTBR's NAT44 init also fails on Talos with
iptables "table does not exist" errors — that's a separate
kernel-module issue that may still block actual Thread border
routing even with the correct radio URL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Synology DSM's UPS server uses source-IP whitelisting as the auth
mechanism — there's no per-user/per-key access control, just an IP
allow-list. Source IPs are transient (k8s-edge eth0 IP, node
migrations, Cilium egress changes) so the integration breaks on
every infra change. Not worth chasing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same dual-run pattern as roomba. Both devices sit on the IoT VLAN,
both are reachable from the pod, but the legacy LXC HA on VM 103 is
still talking to them — so k8s HA hits "Failed to connect" /
TimeoutError on every retry until cutover. Hide the noise via
DEV_MODE; flip false at cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The remote_calendar entry titled "Müllabfuhr" pointed at the public
hostname ical-proxy.renner.uno, which hairpins out through the
Cloudflare tunnel and back into ingress-nginx. Repoint at the
in-cluster Service ical-proxy.apps.svc.cluster.local directly.
Network-policy adjustments to permit the new path:
- home-assistant-ns: add `apps` namespace to egress
- apps/default-deny-ingress: add `home-assistant` to ingress
Both edits stay namespace-scoped (no broader-than-needed selectors)
and follow the existing per-ns CNP pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matter and Z-Wave JS setup were failing with:
Error setting up entry Matter for matter
... _async_ensure_addon_running → get_supervisor_client
Both integrations check `if use_addon:` and call into the HAOS Supervisor
addon manager when true. The entries had use_addon=true and
integration_created_addon=true left over from the VM 103 HAOS install.
On HA Container there's no supervisor → setup crashes.
Fix: extend reconcile_entry with an optional "residue keys" parameter
(matching the MQTT TLS-residue pattern) and pass
["use_addon","integration_created_addon"] for matter and zwave_js.
With those keys gone, HA's early-return path uses the websocket URL
directly.
Also DEV_MODE-disable zwave_js: even with the residue cleaned up, its
deployment is replicas=0 (cutover-pending) so its Service has no
endpoints — connection fails on every retry. Same justification as
edl21 and esphome.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace per-integration DEV_DISABLE_* envs with a single DEV_MODE
switch covering every integration that's blocked until cutover:
- esphome: devices not reachable from the pod network yet
- edl21: stromzaehler USB not mounted yet
- roomba: Roomba MQTT broker is single-client and the legacy HA on
VM 103 still owns the connection — k8s HA gets ConnectionRefused
on every reconnect
Roomba moves from a hard-disable to DEV_MODE-gated; the device IS
reachable from the pod network via the IoT-VLAN egress, the issue is
duplicate-client rejection while we're dual-running. s3_compatible
stays hard-disabled — that's a code-bug (botocore register_feature_id
import), not a cutover-state thing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Quiet four sources of log noise the user surfaced this morning by
keeping their config_entries disabled_by="user":
- s3_compatible: HACS integration imports register_feature_id from
botocore.useragent, removed in newer botocore — fails setup on every
boot. Hard-disabled until the upstream HACS plugin or botocore is
fixed.
- roomba: device on the IoT VLAN unreachable from the pod network in
the current configuration; reconnect loop spams ConnectionRefused.
Hard-disabled until the path is fixed or the device is back online.
- esphome: existing DEV_DISABLE_ESPHOME flag, refactored through the
new disable_entries helper.
- edl21: new DEV_DISABLE_EDL21 flag (default true) for the stromzaehler
SML integration; sml.asyncio spams "/dev/ttyUSB0: No such file" until
the USB is mounted at cutover. Flip to false then.
Adds a disable_entries shell helper next to reconcile_entry. Both
take a jq filter expression so per-domain disables are one-liners.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HAOS-supervisor addon names (core-whisper / core-matter-server /
core-openthread-border-router / core-zwave-js / core-openwakeword /
core-speech-to-phrase) carried over from the VM 103 install but don't
resolve in the cluster. Add a reconcile_entry helper to the sidecar
and use it to repoint each existing config_entry at the matching k8s
Service DNS:
wyoming/Whisper → whisper.home-assistant.svc.cluster.local:10300
wyoming/openWakeWord → openwakeword...:10400
wyoming/Speech-to-Phrase → speech-to-phrase...:10300
matter → ws://matter-server...:5580/ws
otbr → http://otbr...:8081
zwave_js → ws://zwave-js...:3000
Each merge is shallow on a narrow key set (host/port for wyoming, url
for the rest) so HA-managed extras like Z-Wave's s2_*_keys survive
untouched. The helper only patches existing entries; nothing creates
them. Drift detection uses jq -cS canonical key ordering — same
discipline that finally stabilised the MQTT reconcile.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues HA had with the cluster's CiliumNetworkPolicy posture:
1. ingress-nginx-egress's allowlist enumerated apps/authentik/monitoring/
paperless/audiobookshelf/immich/crowdsec but not home-assistant — so
ingress-nginx pods on k8s-1/2/3 were dropped (Policy denied) when they
tried to reach HA at home-assistant.svc.cluster.local. home.renner.uno
was returning 504 gateway timeout because of this. Add home-assistant
to the egress allowlist.
2. The home-assistant namespace had no per-ns CNP, unlike audiobookshelf/
authentik/paperless. Inbound was effectively allow-all. Add
default-deny-ingress modeled on audiobookshelf-ns.yaml, with the
HA-specific extras: ingress from world (IoT devices reach HA via the
Cilium-L2 LB on 10.2.30.224 with source IP preserved), and egress to
authentik + ingress for east-west OIDC, plus world for IoT devices,
external integrations, and the seed tarball pull from garage S3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous merge preserved certificate=auto and tls_insecure=false
left over from the legacy mqtt.renner.uno:8883 config. With the new
plain :1883 broker, HA still tried to negotiate TLS to a non-TLS
listener and got "appears to be using TLS to connect to a non-TLS
listener; protocol error" on every connect attempt — so the entry
was reading correctly but the integration was unusable.
Also treat the *presence* of certificate or tls_insecure in the entry
data as drift, so the residue gets stripped on the next reconcile and
stays gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The drift check compares two jq -c outputs as strings. jq -c
preserves the input's key order, so a filtered current_mqtt has
keys in the source entry's order (broker, password, port, username)
while desired_mqtt has them in the order I wrote them (broker,
port, username, password). Identical values, different bytes,
spurious drift, perpetual self-delete loop.
Add -S to both jq invocations so keys come out sorted lexically and
the comparison reflects values only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous desired_mqtt set included discovery: true and
discovery_prefix: "homeassistant" — both are HA defaults, so HA strips
them on every save of core.config_entries. The sidecar then re-detected
the missing keys as drift, re-patched, self-deleted the pod. New pod
came up, HA stripped defaults again, sidecar drifted again. Pods
cycled in <1m.
Reconcile only fields HA persists: broker, port, username, password.
Defaults are HA's responsibility.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The sidecar gated its sync loop on /config/.HA_RESTORE_RESULT, which
only appears after a HAOS-tarball restore. On a clean install or a
pre-existing /config (the actual current state — HA already has
integrations, including a stale mqtt entry pointing at VM 110), that
flag never materializes and the sidecar blocks forever waiting,
leaving trusted_proxies + MQTT + ESPHome reconciliation never running.
Switch the gate to /config/configuration.yaml — HA writes this on both
paths (post-restore and clean startup), which is what we actually
need anyway since the patch targets that file plus .storage/ entries.
Also: shallow-merge mqtt data instead of replacing it, so
HA-managed fields we don't reconcile (protocol, transport, keepalive,
certificate, tls_insecure, …) survive across patches. Drift detection
narrows to the keys we own so a one-off HA write of e.g. keepalive
doesn't loop us.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues from the previous fix:
1. mosquitto_passwd -c uses O_EXCL, so it errors with "File exists"
on the second container start (the emptyDir at /mosquitto/auth
persists across container restarts within a pod). Add rm -f to
keep the shim idempotent.
2. mosquitto drops privileges to user mosquitto (UID 1883) *before*
reading the password file, so chown root:root left the file
unreadable to the running process. Chown to 1883:1883 instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous shim ran as root (image has no USER directive), wrote
/mosquitto/auth/passwd, chmod'd 0600, then exec'd mosquitto. Mosquitto
opens pwfile/aclfile during startup *before* dropping to user
mosquitto, so the read itself is fine — but the previous code also
left files owned root:1883 due to fsGroup setgid, and used the fragile
mosquitto_passwd -U in-place rewrite path that was failing to open
the file with "Unable to open pwfile".
Switch to the canonical mosquitto_passwd -c -b creation flow and
chown root:root so the startup privilege check stops warning and
the open succeeds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mosquitto runs in the home-assistant namespace, edge-pinned, exposing
plain :1883 + TLS :8883 on a fresh IoT IP (10.2.30.225). HA reaches
the broker cluster-internally; IoT devices land on the IoT IP. TLS
certificate is issued by letsencrypt-staging via DNS-01 (cloudflare),
SANs mqtt.renner.uno + mqtt.k.renner.uno.
Credentials are generated in-cluster on first deploy by a one-shot
Job (idempotent — exits 0 if the Secret already exists), so no
plaintext password ever touches git or a human. The sidecar in the
HA pod (renamed trusted-proxies → config-reconciler to reflect its
broadened scope) reads the same Secret and patches HA's mqtt
config_entry data on drift. MQTT_PASSWORD is marked optional so HA
can come up before the bootstrap Job finishes — the sidecar's MQTT
reconcile skips silently when the env is empty.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copies every dashboard from `ansible/roles/grafana-observability/files/dashboards/`
into `kubernetes/apps/monitoring/dashboards/` and wires each as a
ConfigMap for the kube-prometheus-stack Grafana sidecar. UID rewrites
follow the buckets in DASHBOARD-MIGRATION.md:
- 11 k8s-native dashboards copied with no rewrite (UID stays
`prometheus`/`loki`/`influxdb`).
- 9 host-legacy + 3 mixed-resolved-as-legacy (blackbox, infrastructure,
synology-nas Prom panels) had every `"uid": "prometheus"` rewritten to
`"uid": "prometheus-legacy"` so they continue to render against LXC 227
until the global cut-over retires VM 109.
- log-explorer and netbird-overview are pure Loki, no rewrite.
- paperless-overview and unifi-overview kept on `prometheus` (paperless
per k8s-native intent; unifi because the K8s Prom also scrapes the same
unpoller exporter).
Validated with `kubectl kustomize kubernetes/apps/monitoring`: 27
ConfigMaps generated, all carrying the `grafana_dashboard: "1"` label
that the sidecar watches in the `monitoring` namespace.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds velero/velero-plugin-for-aws to image-automation with a tight
1.13.x range — that matches the velero server major.minor; crossing
to 1.14 should be a deliberate operator decision tied to bumping the
server itself, not an auto-update.
README: catalogues all enrolled images and explicitly lists what's
not enrolled with reasons, so the next person doesn't waste time
trying to wire up tags that can't be policy-tracked (4-component
versions, distro-codename tags, commit-SHA tags, etc.). Cnpg-barman-
plugin is in an upstream-imported manifest and intentionally left
hand-managed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The flux bootstrap was originally run without --components-extra,
so only the four core controllers (source, kustomize, helm,
notification) were installed. The image-* CRDs and Deployments
weren't in the cluster, which meant the image-automation
manifests added in 5c5d39e/b5508d0 immediately broke the
infrastructure Kustomization with "no matches for kind ImagePolicy
in version image.toolkit.fluxcd.io/v1beta2".
Verified by:
kubectl -n flux-system get deployments
-> 4 controllers, no image-*
kubectl get crd | grep image.toolkit
-> empty
kubectl api-resources --api-group=image.toolkit.fluxcd.io
-> empty
Two changes:
1. Regenerated gotk-components.yaml with
`flux install --components-extra=image-reflector-controller,
image-automation-controller --export`.
Now ships all six controllers + their CRDs.
2. bootstrap.sh: same --components-extra flag added to the
`flux bootstrap gitea` call so the next rebuild-k8s run
produces a cluster that includes the image controllers from
the start, not as a one-off bolt-on.
After flux reconciles flux-system, the CRDs land, the new
Deployments come up, and the infrastructure Kustomization can
apply image-automation/ cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original scaffold created a separate flux-system-write GitRepository
with its own security-bot Secret, requiring the user to manually
sops-encrypt and commit a credentials file. That's a manual bootstrap
step — exactly the "needs a human action to reach usable state"
anti-pattern.
`flux bootstrap gitea --token-auth` (kubernetes/bootstrap.sh:202)
already creates the flux-system Secret with a write-capable token.
Verified: `kubectl get secret flux-system -n flux-system -o yaml`
shows username + password fields populated.
Changes:
- update-automation.yaml.sourceRef → flux-system (the existing GR).
- Drop gitrepository.yaml (the redundant flux-system-write CR).
- Drop git-secret.yaml.template and the sops bootstrap dance from
README + kustomization.yaml.
Push identity is now the bootstrap token's owner (gitea_admin) instead
of security-bot. Commit *author* is still overridden to
security-bot@renner.uno via the spec.git.commit.author block, so PRs
and `git log` still surface the bot identity for visibility — only
the actual git push transport uses the bootstrap creds.
Net result: zero manual steps. Push this and image-automation is live.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds kubernetes/infrastructure/image-automation/ — central home for
Flux image-automation resources. The image-reflector + image-automation
controllers are already running (gotk-components.yaml ships them);
this is the first time we declare CRs that use them.
What lands:
- flux-system-write GitRepository (separate from the bootstrap
flux-system GR so we don't disturb its existing read-only auth)
pointing at gitea.renner.uno/gitea_admin/homelab via a Secret
named image-automation-git.
- ImageUpdateAutomation "homelab" walking ./kubernetes, applying
the Setters strategy from `# {"$imagepolicy": "..."}` markers,
pushing direct to master as security-bot@renner.uno.
- Pilot pair: ImageRepository + ImagePolicy "paperless-ngx" tracking
ghcr.io/paperless-ngx/paperless-ngx within "2.x" semver. The 3.x
cutover would need manual policy widening — by design.
- Marker on kubernetes/apps/paperless/paperless.yaml's image line
so the controller knows where to substitute.
- README explaining the bootstrap step and how to onboard more
images.
Manual one-time bootstrap REQUIRED before this works:
1. Create kubernetes/infrastructure/image-automation/git-secret.sops.yaml
from the template (token = security-bot Gitea PAT, write:repository
scope — same one used by the Woodpecker security_bot_token secret
can be reused).
2. sops -e -i to encrypt in place.
3. Uncomment the line in kustomization.yaml that lists it.
4. Push. Flux's existing infrastructure Kustomization handles the
sops-decryption.
Strategy choice (per user): direct push to master, not a PR branch.
Tight semver ranges per image are the safety net.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Container image tag updates are Flux's job — the homelab cluster
already runs image-reflector-controller and image-automation-
controller (in flux-system/gotk-components.yaml). Setting up
ImageRepository / ImagePolicy / ImageUpdateAutomation resources
is the right home for "bump postgres:15.17 → 15.18 when a new
image is published"; doing it from a CVE-driven scan agent
duplicates that pipeline and competes with it.
Reverting:
- Agent prompt: drop the `## Image tag bumps` section; image
findings now go straight to the Skipped list with a "Flux
image-automation owns this" reason.
- Woodpecker yml: drop Bash(crane *) from the agent's tool
allowlist (no longer used) and the `pull: true` workaround
that existed only to pick up crane's mid-day rebuild.
- ci-claude Containerfile: remove the crane install. The image
is back to Node + Claude Code + git/jq/curl basics.
The agent is now scoped exclusively to IaC misconfigurations
(Trivy fs scan results). Image vulns surface in the log + summary
for visibility but no PR action.
Note: there are currently zero ImageRepository / ImagePolicy /
ImageUpdateAutomation resources in kubernetes/ — the controllers
are running idle. Wiring them up for the actual images is a
separate workstream.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Velero CSI snapshot data movement against an RWO ceph-block (RBD) PVC
fails when the source pod still has it RW-mounted on the same node:
mount: cannot mount /dev/rbdN read-only
The krbd device can't be active in two modes simultaneously, so the
snapshot exposer pod's RO mount conflicts with the writer pod's RW
mount. Same failure mode is documented in
kubernetes/seed/music-assistant/pvcs.yaml.
Switch the test PVC to ceph-filesystem (CephFS RWX) — concurrent RO
snapshot mount + RW source mount are fine on CephFS. The test still
exercises the same Velero CSI data-movement code path end-to-end
(snapshot → upload to BSL → restore → mount).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the seed-phase refactor (3063b54), authentik-db / paperless-db
Kustomizations no longer exist — they were merged into authentik /
paperless KSs (commit ae76682), then the database resources moved
into the new seed KS. The e2e-test was still grepping for the old
names and FAILing on every run. Drop them and add the new seed KS
to the readiness check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 864's agent reported crane was not installed even though
the build-claude step had baked crane into ci-claude:2026-05-01.
The wp-agents had the pre-crane image cached at the same tag —
woodpecker reuses already-pulled images by default — so the agent
was running on a stale layer set.
pull: true tells woodpecker to refresh the image on every run,
which surfaces same-day rebuilds (the date tag stays put while
contents change). Once the tag rolls over for a different reason
this can come back off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rebuild-k8s pipeline previously SOPS-encrypted talosconfig and
kubeconfig and pushed them back to master. The git push raced with
concurrent commits and forced workstation users to `git pull --rebase`
after every rebuild.
Replace the transport with Garage's k8s-secrets bucket
(s3://k8s-secrets/cluster-access/), reusing the existing secrets-key
that the install-garage-synology playbook already provisions. SOPS
encryption is unchanged — only the storage layer moves.
- New scripts/ci/fetch-kubeconfig.sh helper: 9 CI steps that previously
inlined `sops -d kubernetes/access/kubeconfig.sops` now call this.
- scripts/import-cluster-access.sh rewritten to pull both files from
Garage; no more git fetch/checkout.
- export-access step uploads via aws s3 cp; gitea_token + git config
removed.
- All consuming Woodpecker steps gain garage_secrets_key_id /
garage_secrets_secret_key from_secret env vars, and bump to
ci-k8s:2026-05-01 (the tag with aws-cli baked in).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 861's agent step ran claude successfully but the workflow
ended in failure with `/bin/sh: syntax error: unterminated quoted
string` — emitted between claude's output and the next command
block's tracer.
Cause: the prior pattern was
claude ... -p "$(cat ci/agents/security-fix/prompt.md)"
which substitutes a ~6 KB blob (lots of backticks, em-dashes, code
snippets, contractions) onto the command line. Under woodpecker's
busybox-ash invocation with `set -ex`, the tracer mishandled the
substituted argument and aborted the script — even though the
underlying claude call already finished.
Fix: pipe the prompt file via stdin instead. claude `-p` reads
the prompt from stdin when no positional argument is given (verified
locally against the prepared image — STDIN_OK round-trip).
Other fallout: removed em-dashes and backticks from the surrounding
shell comments to keep the trace output unambiguous, and added a
note explaining why stdin is used.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The seed KS owns influxdb-seed-ssh Secret in the influxdb namespace, but
the namespace was previously created by apps/influxdb/namespace.yaml —
which dependsOn seed → seed never reconciles → circular.
Hoist the namespace into infrastructure/namespaces/namespaces.yaml
(consistent with all the other app namespaces) and drop it from the
apps/influxdb Kustomization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 858's agent run found 277 image vulns (CRITICAL/HIGH) and
correctly skipped all of them — but the right action for many is a
patch-level image-tag bump (postgres:15.17 → 15.18,
distribution:3.0.0 → 3.0.1, etc.). The agent had no way to verify
a candidate tag exists in the registry, so it conservatively
skipped.
crane (google/go-containerregistry) handles:
crane ls <repository> — list available tags
crane manifest <image>:<tag> — verify tag resolves
Single static binary, works against Docker Hub, GHCR, gitea, quay,
any OCI registry. Pinned to v0.21.5.
The agent prompt update (only-patch bumps, lockstep cross-file edits,
compatibility-caveat surfacing) and the woodpecker allowlist update
(Bash(crane ls *), Bash(crane manifest *)) ride in a follow-up
commit so this image change can build first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the per-app race-y CNPG bootstrap.recovery + velero-bootstrap-restore
pattern with a single seed Flux KS that populates databases and PVCs from
legacy hosts before any app KS reconciles.
Changes:
- New kubernetes/seed/<app>/ tree owning, per app:
- CNPG Cluster CRs with bootstrap.initdb (was: bootstrap.recovery) +
enableSuperuserAccess so seed Jobs can connect as postgres
- DB seed Jobs: ssh legacy 'pg_dump -Fc' | pg_restore -d <db> against
the running cluster (was: ephemeral PG + barman-cloud-backup → S3)
- PVC seed Jobs: rsync/ssh+tar from legacy → app PVCs as standalone
Jobs (was: in-pod init containers, blocking the app Deployment's
progressDeadlineSeconds for hours on a 250 GiB rsync)
- All ObjectStores, ScheduledBackups, barman-creds, PVCs, seed-ssh
Secrets relocated here
- New top-level kubernetes/clusters/homelab/seed.yaml Flux KS:
dependsOn infrastructure-cnpg-barman, timeout 4h, healthCheckExprs
cover CNPG Cluster phase + native Job succeeded.
- Stripped from kubernetes/apps/<app>/: Cluster CRs, seed Jobs, init
container rsync blocks, ssh volumes, seed ServiceAccount references.
~454 lines deleted, ~33 added.
- App KSs (immich, authentik, paperless, audiobookshelf,
music-assistant, influxdb): dependsOn velero-bootstrap-restore →
dependsOn seed.
- bootstrap.sh wait-apps: new wait_kustomization seed (4h) before
per-app waits; per-app timeouts dropped to 60s since they're
near-instant once seed is Ready.
- Deleted kubernetes/clusters/homelab/velero-bootstrap-restore.yaml;
manifests stay in kubernetes/infrastructure/velero-bootstrap-restore/
for post-cutover re-add (it remains the cold-rebuild path once
legacy hosts are decommissioned and seed phase is removed).
Eliminates the documented race failure modes:
- cnpg-bootstrap-recovery-no-initdb-fallback (recovery Job pounds empty
bucket until seed lands → multiple Error pods)
- velero-bootstrap-restore-bsl-sync-race (Job iterates schedules in an
empty bucket for ~15 min)
- immich-server progressDeadlineSeconds blowup (250 GiB rsync init
exceeds the 600s default and Flux marks the Deployment Failed)
Seed is migration-only: removed entirely once all services have cut
over from legacy. At that point the Cluster + ObjectStore +
ScheduledBackup + PVC manifests move back into kubernetes/apps/<app>/
and velero-bootstrap-restore.yaml is re-added at the cluster level.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 856's agent run found the Trivy JSON files via the
pipeline's own jq summary (IaC=0, Image=645 findings) but the
agent itself reported "No Trivy scan files (/tmp/trivy-iac.json
...) were present" — it hallucinated the /tmp prefix and didn't
look in the workspace root.
Prompt now states explicitly that the JSONs are in the current
working directory (project root, not /tmp), and instructs the
agent's first action to be `Glob trivy-*.json` to enumerate what
exists. Adds a stop-condition for the genuinely-empty case so
the agent can't get into a "I'll assume they're somewhere else"
loop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Earlier I added `status: [success, failure]` to the agent's `when`
under the assumption that image-scans's `failure: ignore` wouldn't
prevent downstream skipping. That was correct in mechanics but
wrong in intent: the agent SHOULD only run when the scans actually
completed cleanly. If iac-scan or clone fails, the agent has nothing
trustworthy to act on — better to surface the workflow failure
than open a PR based on partial/missing JSON.
`failure: ignore` on image-scans is kept (a single bad image
shouldn't block the run); other failure modes correctly cascade
to skip the agent.
Pipeline 853's "skipped agent" outcome was actually a Woodpecker
glitch (clone retroactively flipped to failure mid-run) and not a
design problem — once that glitch passes, default success-only
gating is the right behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 850 reproduced two distinct failures:
1. image-scans died — Apache Tika hit Trivy's default 5-minute
`context deadline exceeded`, and goauthentik/server's 30MB
msgraph kiota-dom-export.txt blew the secret scanner's memory
budget (Trivy itself warned about it).
2. security-fix-agent was skipped despite `failure: ignore` on
image-scans. Woodpecker's `failure: ignore` only affects
*workflow* status — downstream steps still gate on the
previous step's exit, defaulting to status: [success].
Fixes:
- `--timeout 15m` and `--scanners vuln` on per-image trivy calls.
We only care about CVEs (CRITICAL/HIGH, with FixedVersion) — the
secret scanner was the OOM culprit and adds no value for our
agent's decision rules.
- `when: status: [success, failure]` on security-fix-agent so it
always runs, regardless of which scans crashed. The step's own
gating logic already handles missing/empty JSON files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.7
security-bot