Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:
- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
ci-diagnostics with a read-only ClusterRole (workloads, events,
pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
plus a namespaced services/proxy Role for the Prometheus and
Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
the admin kubeconfig (no new long-lived secret), builds
.ci-diag/kubeconfig, copies the static kubectl binary for the
jq-only ci-claude image, smoke-tests with the restricted identity,
and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
hypothesis you could have settled with one read-only command but
didn't is a defect in your output'; fix agent must test the issue's
claims before building on them; minimal-quote rule against dumping
logs into public issues.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
rebuild-k8s.yml was the one-shot PVE->K8s migration/rebuild pipeline; it
is no longer needed and still pointed at the retired Garage S3 on the
now-off nas-1. Remove it (DR restore is via Velero + flux bootstrap).
The live cluster-access path is kept and migrated off Garage onto Hetzner
Object Storage (renner-uno-k8s-secrets, fsn1.your-objectstorage.com,
region fsn1, hetzner_s3_* keypair):
- scripts/ci/fetch-kubeconfig.sh: Garage -> Hetzner; env vars renamed
GARAGE_SECRETS_* -> S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY.
- .woodpecker/e2e.yml, e2e-test.yml: pass hetzner_s3_access_key/secret_key
(the live cron e2e pipeline needs this — it was broken against dead nas-1).
- scripts/import-cluster-access.sh: Garage -> Hetzner endpoint/bucket/region.
NOTE: the workstation aws-cli "secrets" profile must now hold the Hetzner
Object Storage keypair.
Also: delete the orphaned test-garage-smoke play + pipeline, and fix the
two dangling rebuild-k8s.yml references (zfs-tuning play comment;
update-audit prompt Talos-pin file list).
Assumes cluster-access objects (kubeconfig.sops/talosconfig.sops) were
migrated into renner-uno-k8s-secrets; with rebuild-k8s gone, nothing
auto-refreshes them, which is fine for a stable cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Revert PR #19: the custom worktree wrapper scripts duplicated Claude
Code's native worktree support and are not the intended workflow.
Use the built-in worktree feature instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
claude-start.sh creates (or reuses) a sibling git worktree at
../homelab-<name> on a new branch off master and launches claude in it.
claude-stop.sh tears the worktree down, refusing on uncommitted changes
unless --force, with optional --delete-branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The rebuild-k8s pipeline previously SOPS-encrypted talosconfig and
kubeconfig and pushed them back to master. The git push raced with
concurrent commits and forced workstation users to `git pull --rebase`
after every rebuild.
Replace the transport with Garage's k8s-secrets bucket
(s3://k8s-secrets/cluster-access/), reusing the existing secrets-key
that the install-garage-synology playbook already provisions. SOPS
encryption is unchanged — only the storage layer moves.
- New scripts/ci/fetch-kubeconfig.sh helper: 9 CI steps that previously
inlined `sops -d kubernetes/access/kubeconfig.sops` now call this.
- scripts/import-cluster-access.sh rewritten to pull both files from
Garage; no more git fetch/checkout.
- export-access step uploads via aws s3 cp; gitea_token + git config
removed.
- All consuming Woodpecker steps gain garage_secrets_key_id /
garage_secrets_secret_key from_secret env vars, and bump to
ci-k8s:2026-05-01 (the tag with aws-cli baked in).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Delete kubernetes/apps/authentik/postgres.yaml: orphan CNPG Cluster with
bootstrap.initdb and no plugin declaration; would compete with the
working ak-pg in authentik-db/ if accidentally applied.
- Delete kubernetes/apps/owncloud/: subtree never wired into Flux (no
Kustomization in clusters/homelab/ targets it), would not deploy.
- Drop CLAUDE.md: superseded by ~/.claude/CLAUDE.md global instructions.
- rebuild-k8s.yml: track per-module retry failures and exit 1 if any
module didn't recover; add set -e to disk-wipe step. Stops masking.
- import-cluster-access.sh: drop "|| true" on git checkout so missing
access files surface immediately instead of silently continuing.
- Add scripts/trust-letsencrypt-staging.sh: helper to install Let's
Encrypt staging roots into system or Firefox trust stores.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the OBC CEL health check lived on infrastructure-ceph-cluster,
which holds OBCs for every app. A slow bind in one namespace stalled
every other app that depends on infrastructure-ceph-cluster. Move each
OBC into its own app Kustomization (paperless-db, authentik-db) and
apply the health check there so each app only waits on its own bucket.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rebuild pipeline commits a new talosconfig/kubeconfig each run.
Without a git pull beforehand, import-cluster-access imports stale
configs and kubectl fails with x509 CA mismatch against the newly
rebuilt cluster.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/import-cluster-access.sh decrypts the SOPS-encrypted
talosconfig and kubeconfig from kubernetes/access/ into ~/.talos/
and ~/.kube/. Run after a rebuild-k8s pipeline completes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Downloads backup from RustFS S3 directly into the CNPG pod using
the MinIO client. No extra pods or complex orchestration needed.
Requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backup pipeline (CI): runs pg_dump via Ansible on the old DB server,
uploads dump to s3://migration/ bucket in RustFS. No extra secrets needed.
Restore script (workstation): downloads from S3, copies into K8s
CNPG pod, runs pg_restore. Uses local kubectl and aws-cli.
Preserves all users, passwords, 2FA, WebAuthn, and sessions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backup pipeline (CI): runs pg_dump via Ansible on the old DB server,
uploads dump to RustFS S3. No kubeconfig secret needed.
Restore script (workstation): downloads from S3, copies into K8s
CNPG pod, runs pg_restore. Uses local kubectl.
Preserves all users, passwords, 2FA, WebAuthn, and sessions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- set_password endpoint returns empty body on success
- Remove --silent flag from flux bootstrap (unsupported)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bootstrap now waits for the authentik kustomization after apps.
Include the Authentik API migration script for post-bootstrap setup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
claude-botandClaude Fable 5
k.rennerandClaude Opus 4.8