Commit Graph
4354 Commits
Author SHA1 Message Date
claude-botandClaude Opus 4.8 f9c5ab6b1b feat(ci): trigger pr-review on pull_request events instead of hourly cron
Switch the PR review bot from a homelab-hosted hourly cron that polls four
repos over the Gitea API to a per-repo, event-driven pipeline that reviews
only the PR that triggered it (CI_REPO + CI_COMMIT_PULL_REQUEST). Fires on
pull_request, which Woodpecker raises on both PR-open and push-to-open-PR.

Drops the cross-repo discovery loop and MAX_PRS cap; keeps the sha-dedup
marker (now guarding pipeline retries), Haiku triage, model selection, and
never-approve posture. Resolves the agent prompts from the local checkout or
a shallow homelab clone so the identical file drops into every reviewed repo.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 09:13:37 +02:00
k.renner ec78cad10a Merge pull request 'Hairpin NAT for NetBird proxy addresses + public-DNS cutover for roaming clients' (#361) from claude-bot/netbird-proxy-2-hairpin into master
ci/woodpecker/push/deploy-infrastructure Pipeline was successful
ci/woodpecker/cron/pr-review Pipeline was successful
ci/woodpecker/push/deploy-netbird-proxy Pipeline was successful
ci/woodpecker/manual/deploy-infrastructure Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/manual/pr-review Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#361
2026-07-23 07:11:32 +00:00
claude-botandClaude Fable 5 4806e55617 fix(dns): k.renner.uno stays out of domainFilters, no exceptions
Owner directive: .k is never reintroduced, not even as a temporary
garbage-collection scope. The leftover k. records on the UDM are out of
external-dns's view and get deleted once via the UDM UI at cutover.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
2026-07-23 09:09:26 +02:00
claude-botandClaude Fable 5 757cd828a4 fix(dns): keep k.renner.uno filter as temporary garbage collector
Review finding: domain filters gate deletion as well as creation, so
dropping the suffix would orphan the retired k. records instead of
removing them. The filter entry stays until sync has deleted them all
(no resource carries a k. host, so desired state for the suffix is
empty), then gets dropped — the comment says exactly when. Also fixes
the stale excludeDomains mention and the '13 networks' miscount in the
NAT header comments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
2026-07-23 09:08:00 +02:00
claude-botandClaude Fable 5 239151b6b4 fix(dns): purge every remaining .k.renner.uno reference
*.k.renner.uno was a temporary migration namespace and is banned (owner
directive 2026-07-14, documented in openbao/ingress.yaml) — it was never
the live naming. Removes the .k. hosts/SANs from ical-proxy, loki,
grafana and the mosquitto cert, renames syslog.k.renner.uno to
syslog.renner.uno (cert + external-dns annotation; any syslog sender
still configured with the k. name must be repointed), fixes the stale
comments, updates e2e-test.sh probes to the real names, and drops
k.renner.uno from the external-dns domainFilters so leftover k. records
get garbage-collected by policy: sync.

The migration script's historical docstring reference stays — it
documents what the migration did.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
2026-07-23 08:55:05 +02:00
claude-botandClaude Fable 5 b894fdb11d fix(dns): label filter needs no manual record cleanup
Correction: only domain filters apply to both desired and current
state. The label filter is source-side only, so the registry-owned UDM
records for newly labeled resources stay visible and policy: sync
deletes them itself — the cutover is fully automatic.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
2026-07-23 08:50:29 +02:00
claude-botandClaude Fable 5 ca61ff493c feat(dns): label-driven registration replaces the excludeDomains list
The dns.renner.uno/public=true label on an Ingress marks it as publicly
registered (Cloudflare wildcard -> NetBird proxy); external-dns-unifi
now runs with labelFilter dns.renner.uno/public!=true and skips those
resources entirely. Visibility is a property of the app resource itself
— there is no central name list to keep in sync with the proxy.

Also removes the dead .k.renner.uno hosts from the public apps'
Ingresses (their TLS certs reissue without those SANs) and splits
photos.renner.uno onto its own unlabeled Ingress: the label filter is
per-resource and that name stays LAN-registered. Remaining .k. names on
LAN-only apps (loki, ical-proxy, syslog) are untouched here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
2026-07-23 08:38:17 +02:00
claude-botandClaude Fable 5 8003f6a1ef fix(opentofu): no hairpin on VLANs with cluster-node legs; tidy
Review finding: on (40) Server, (42) Ingress and (49) k8s Ceph the
cluster nodes sit on the same L2 as the client, so a DNATed flow's
reply bypasses the UDM conntrack (asymmetric) — and clients there are
the nodes themselves, which already resolve the public names in-cluster.
Their idx slots stay reserved. Also records the restapi provider in the
lock file and trims trailing blank lines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
2026-07-23 08:20:05 +02:00
claude-botandClaude Fable 5 a7ab5c6b27 docs(external-dns): excluding a domain hides records, does not delete them
Review finding: the domain filter applies to current state as well, so
pre-existing LAN records for newly excluded names are orphaned on the
UDM and must be removed once by hand as part of the cutover.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
2026-07-23 08:09:30 +02:00
claude-botandClaude Fable 5 c74966808b refactor(opentofu): uniform hairpin rule shape for both address families
Per review: no family-specific behavior. Every rule is the same
ADDRESS_AND_PORT specific-address shape (endpoint x port x network,
6 rules per network), matching the pre-existing hand-made rules'
structure. Drops the tofu-owned firewall groups and the v4-only
FIREWALL_GROUPS path entirely.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
2026-07-23 07:56:25 +02:00
claude-botandClaude Fable 5 1e6ed460ed feat(opentofu): hairpin DNAT rules for NetBird proxy addresses as code
No UniFi Terraform provider models v2 NAT translation fields
(ubiquiti-community 0.55 none, filipowm #104 open, resnickio shell-only),
so the rules go through Mastercard/restapi against the controller's
/proxy/network/v2/api/site/default/nat endpoint, same API key as the
unifi provider.

Rule shapes follow the behaviorally verified matrix on Network 10.5.62:
v4 uses FIREWALL_GROUPS (tofu-owned address+port groups); v6 must use
ADDRESS_AND_PORT specific-address rules (4 per network) because v6
group-based DNAT silently drops the address match and rewrites all
traffic on the listed ports. rule_index is static per network to keep
indices collision-free and stable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
2026-07-23 07:49:04 +02:00
security-bot 16c1ea1d04 chore(image-automation): bump gitea.renner.uno/k.renner/vsftpd:build.1784731469
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/pr-review Pipeline was successful
Updates managed by image-automation-controller.
2026-07-22 14:57:57 +00:00
k.renner ed567f300a Merge pull request 'feat(woodpecker): upgrade to v3.16.0 + enable 3.x image automation' (#359) from claude-bot/woodpecker-316-imgauto into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#359
2026-07-22 14:38:14 +00:00
security-bot 463f9e59be chore(image-automation): bump gitea.renner.uno/k.renner/vsftpd:build.1784729100
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-07-22 14:27:57 +00:00
claude-botandClaude Opus 4.8 511904ae9e feat(woodpecker): upgrade to v3.16.0 + enable 3.x image automation
Now that woodpecker runs on postgres (not the previously-pinned sqlite), the
"never surprise-bump a live sqlite DB" hazard is gone. Bump server + agent
3.13.0 -> 3.16.0 (a reviewed minor jump: no breaking config changes for our
kubernetes+gitea+postgres setup, no risky migrations per the changelog; direct
3.13->3.16 is supported, migrations run on startup).

Add Flux image automation (image-automation/woodpecker.yaml) at range 3.x, so
minor + patch releases auto-update; only the 4.0 major stays a reviewed bump.
Server and agent share the same range/tag-filter and both carry an $imagepolicy
marker, so they bump in lockstep (gRPC version parity is required).

Pre-upgrade CNPG base backup taken (wp-pg-preupgrade-316) as the rollback path,
since migrations are forward-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 16:24:27 +02:00
k.renner 52a36336f1 Merge pull request 'feat(woodpecker): drop the sqlite PVC, StatefulSet->Deployment (#295, #324)' (#357) from claude-bot/woodpecker-drop-pvc into master
ci/woodpecker/cron/pr-review Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#357
2026-07-22 14:11:37 +00:00
claude-botandClaude Opus 4.8 709854c6b4 feat(woodpecker): drop the sqlite PVC, StatefulSet->Deployment (#295, #324)
Stage 3/3, the finale of the sqlite->CNPG migration. woodpecker-server is now
stateless (DB in wp-pg Postgres, verified end-to-end incl. a live pipeline run),
so convert the StatefulSet to a Deployment and remove the volumeClaimTemplates +
/var/lib/woodpecker mount. strategy: Recreate keeps a single control-plane
instance (no two servers on one DB during a roll).

This deletes the last use of the data-woodpecker-server-0 PVC, whose ext4
CSI-snapshot backup failed permanently ("structure needs cleaning"). Once the
PVC itself is removed, Velero's apps-daily stops trying to snapshot it —
resolving #295 and #324. The agent reaches the server via the standalone
`woodpecker` Service (label selector), so a Deployment pod is reachable
unchanged.

Closes #295
Closes #324

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 16:10:25 +02:00
k.renner 4fbd437e78 Merge pull request 'feat(woodpecker): cut server over to wp-pg postgres (#295, #324)' (#356) from claude-bot/woodpecker-pg-cutover into master
Reviewed-on: k.renner/homelab#356
2026-07-22 14:02:18 +00:00
claude-botandClaude Opus 4.8 7849b6bb64 feat(woodpecker): cut server over to wp-pg postgres (#295, #324)
Stage 2/3 of the sqlite->CNPG migration. Points woodpecker-server at the
wp-pg Postgres cluster (creds from the CNPG-generated wp-pg-app secret via a
lib/pq keyword DSN). The sqlite data was migrated one-time via pgloader
(schema-first/data-only, per seed/woodpecker/MIGRATION.md); parity verified
(all 18 tables match, booleans correct, sequences reset).

The sqlite PVC + volumeMount are intentionally KEPT for now (rollback = revert
these env vars) — they're dropped in stage 3 (StatefulSet->Deployment) after a
soak. Image stays v3.13.0 (no version bump during the migration).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 15:56:45 +02:00
claude-botandClaude Opus 4.8 66c2f8d9ea fix(external-dns): stop publishing LAN-only records for public hostnames
A phone that caches the LAN answer for a public app on wifi (10.2.99.86 /
fd99::86) and then roams to cellular blackholes on it: RFC1918 and ULA give
no RST and no ICMP, so the app hangs until TCP timeout instead of failing
fast. That is why restarting the app does not always help — the OS resolver
still hands back the unroutable address.

Excluding these eight names from the UniFi provider makes LAN fall through to
the public *.renner.uno wildcard, so every network returns the same answer and
a stale cache stays correct. LAN traffic still stays local: the UDM hairpins
the proxy addresses back to 10.2.99.86 / fd99::86 (Policy Table -> NAT).

Excluded by record name rather than by label because these Ingresses are
multi-host — immich.k.renner.uno and immich.renner.uno share one object, and
*.renner.uno cannot cover a three-label name, so a label filter would make the
.k. name unresolvable on LAN.

policy: sync, so this deletes 16 UDM records (8 names x A/AAAA) on reconcile.

The source of truth for the list is the NetBird proxy's tunnelled domains, not
the cluster; it must be updated when an app is published or unpublished.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 15:45:46 +02:00
k.renner 61b1f7b64d Merge pull request 'feat(woodpecker): provision wp-pg CNPG cluster to migrate off sqlite (#295, #324)' (#355) from claude-bot/woodpecker-cnpg into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#355
2026-07-22 13:30:07 +00:00
claude-botandClaude Opus 4.8 5cb90432e7 feat(woodpecker): provision wp-pg CNPG cluster to migrate off sqlite (#295, #324)
woodpecker-server runs sqlite on a ceph-block PVC (data-woodpecker-server-0)
whose Velero CSI-snapshot backup fails permanently (ext4 "structure needs
cleaning" — the source fs is error-flagged and journal replay can't clear it).
Rather than repeatedly fsck a fragile sqlite, move woodpecker's DB onto CNPG
postgres like authentik/immich/monitoring/paperless already do — giving proper
WAL-based barman backups and letting the PVC be removed entirely, which
resolves #295/#324 by deletion.

This PR (stage 1 of 3) provisions only the empty backend — no woodpecker
impact:
- kubernetes/seed/woodpecker/: wp-pg Cluster (1 instance, initdb woodpecker),
  ObjectStore -> Hetzner S3 (s3://renner-uno-k8s-cnpg/wp-pg), ScheduledBackup,
  and a hetzner-s3-creds ExternalSecret (reuses the shared paperless S3 path to
  avoid seeding a new OpenBao key; follow-up noted inline to give it its own).
- seed-woodpecker Flux Kustomization (dependsOn infrastructure + cnpg-barman,
  CNPG health-gated), auto-synced from clusters/homelab/.
- MIGRATION.md: the one-time schema-first / data-only pgloader runbook and the
  staged cutover (PR 2 = server.yaml -> postgres; PR 3 = drop PVC, STS->Deploy).

sqlite PVC is untouched; rollback is reverting env vars until PR 3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 15:28:37 +02:00
k.renner f14fb80589 Merge pull request 'fix(flux): scope infrastructure health to its operators, not every resource' (#354) from fix/infra-healthchecks into master
Reviewed-on: k.renner/homelab#354
2026-07-22 13:26:25 +00:00
claude-botandClaude Opus 4.8 93944ff525 fix(flux): scope infrastructure health to its operators, not every resource
The infrastructure Kustomization used wait: true with no healthChecks. With
wait: true Flux ignores healthChecks and blocks the Kustomization's Ready on
EVERY resource it applies (hundreds). Because ~28 kustomizations dependsOn
infrastructure, one crashlooping workload froze the entire GitOps apply chain:
a crowdsec-agent crashloop held all of GitOps hostage for hours — merges didn't
go live — even though crowdsec is a non-critical app nothing depends on for
startup. The 15m health-check timeout looped every 10m, so infrastructure never
reported Ready and every dependent stayed blocked.

Switch to wait: false + explicit healthChecks covering only the operators and
providers dependents actually consume: cilium (CNI), cert-manager (TLS Secrets),
external-secrets + openbao (the ESO backend ExternalSecrets resolve against),
rook-ceph-operator (storage), cloudnative-pg (DB operator for seed jobs).
Everything else still applies — it just no longer gates readiness, so a
non-critical crashloop can't cascade across the cluster. This mirrors the
pattern infrastructure-ceph-cluster already uses (healthCheckExprs for Rook).

Verified live: infrastructure reconciles Ready on the 6 checks (wait=false),
and all dependent kustomizations stayed Ready.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:59:15 +02:00
claude-botandClaude Opus 4.8 3984cd0d14 feat(netbird-proxy): add proxy-2 member, restore port 80, stop needless restarts
Adds proxy-2.renner.uno (netcup, Arch) as a second proxy-only cluster member
so proxy.renner.uno can move off netbird.renner.uno entirely. The controller
host shared one IP and port with the reverse proxy, separated only by TLS SNI,
which made destination NAT for LAN hairpin impossible without also capturing
NetBird control-plane traffic.

netbird-proxy listens on :443 only (NB_PROXY_ADDRESS=:443) and has no HTTP
listener, so once proxy.renner.uno stopped resolving to the controller host,
port 80 went unanswered for every off-LAN client. Adds a minimal nginx
redirector publishing :80 on every member to restore the 301.

Deploying that sidecar exposed two defects in the role:

- Any docker-compose.yml change notified the force-recreate handler, rebuilding
  the proxy container even for edits that do not touch the proxy service.
  `docker compose up -d` already recreates only changed services, so the notify
  is dropped there; proxy.env keeps it because compose cannot detect env_file
  content changes.
- The play restarted every cluster member in parallel, taking the whole proxy
  cluster down at once. Now serial: 1.

Verified: both members serve :443 (200) and :80 (301); a repeat run reports
changed=0 on both with no handler firing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 14:54:05 +02:00
k.renner ab66e4c67e Merge pull request 'feat(flux): auto-rotate the Flux git token (gitea-token mode)' (#353) from feat/flux-token-rotation into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/pr-review Pipeline was successful
Reviewed-on: k.renner/homelab#353
2026-07-22 11:54:39 +00:00
claude-botandClaude Opus 4.8 9f43ad7abb chore(flux): pin GITEA_REQUIRE_PUSH=true on the flux-git rotator
Per review. k.renner/homelab is public, so the pull verification passes for any
valid token — push is the only load-bearing check. It already defaulted to true
in the binary, but pinning it in the manifest documents the safety invariant at
the call site so it can't silently degrade if the default ever changed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 13:49:44 +02:00
claude-botandClaude Opus 4.8 0d6fc81e9b feat(flux): auto-rotate the Flux git token (gitea-token mode)
Wires the Flux git credential — the dedicated `flux` gitea account — into the
rotation fleet, so it can't silently die again (the cause of the recent GitOps
outage). Quarterly, via the secret-rotator binary's gitea-token mode.

Flow: mint a new PAT (basic auth, the flux password bootstrap) -> VERIFY it can
pull AND push (Flux image-automation writes back) -> store in OpenBao -> ESO
merges it into the flux-system secret -> source-controller reads it fresh. A
token that fails verification is deleted and the run aborts without storing, and
the reap keeps the two newest tokens so the previous one stays valid through the
ESO propagation window — Flux is never stranded.

Reachability: gitea via the in-cluster Service gitea.apps.svc:3000 — the public
gitea.renner.uno name hairpins off the ingress LB from a pod (same as
pve.renner.uno). Needs both netpol halves (gitea-allow-flux-rotator in apps +
rotator-gitea-flux-egress in openbao). Added flux-system to the openbao
ClusterSecretStore's allowed namespaces so the ExternalSecret can sync there.
The ExternalSecret uses creationPolicy: Merge, so an ESO failure leaves the
last-good credential in place rather than blanking Flux's secret.

Tested live end-to-end: job minted flux-gitops-1b8d20, verified pull+push,
stored (OpenBao e26e8bf1 -> 9666320e), kept the previous token (2 tokens, 0
reaped), ESO synced, Flux pull fetched, and image-automation pushed
(Ready=True). OpenBao policy/role + the gitea-flux password bootstrap are
provisioned by hand (see rotator-gitea-flux.hcl).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 12:57:25 +02:00
k.renner f5fa8f5a1d Merge pull request 'feat(secret-rotator): gitea-token mode' (#348) from feat/gitea-token-mode into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/pr-review Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#348
2026-07-22 10:48:01 +00:00
claude-botandClaude Opus 4.8 59e804d9a4 fix(secret-rotator): keep the previous gitea token through the ESO propagation window
ci/woodpecker/push/build-ci-images Pipeline was successful
Second review pass (no blocking issues). The important one: the reap deleted
EVERY other prefix token right after store, including the previous run's token —
which is still the live credential in the k8s Secret until ESO syncs the new
value and source-controller re-reads it. Deleting it immediately opens an
availability window (ESO refreshInterval + reconcile lag) where Flux's stored
token has just been revoked. Since this mode exists specifically to avoid
stranding Flux, that window shouldn't exist.

Now the reap keeps the TWO newest tokens (the one just stored + the previous)
and deletes only older ones. tokens_with_prefix returns (id, name) so the
caller can order by recency; the previous token survives until the next
quarterly run, long after ESO has synced.

Also:
- doc-note that PUSH is the load-bearing gate check: for a public verify_repo,
  git-upload-pack returns 200 for any valid token, so pull barely gates —
  don't set GITEA_REQUIRE_PUSH=false against a public repo.
- fetch tokens with limit=50 (single effective page) and note the count stays
  tiny given keep-two reaping, so pagination can't silently miss old tokens.
- tidy the cleanup error message connector.

clippy -D warnings clean; 14 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 12:45:07 +02:00
k.renner 84d94186ec Merge pull request 'fix(#336): stop the registry filling — frequent GC + stop needless CI rebuilds' (#350) from claude-bot/gitea-registry-gc into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/pr-review Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#350
2026-07-22 09:03:59 +00:00
claude-botandClaude Opus 4.8 f6bee49e6d fix(gitea): set volumeClaimTemplate to 40Gi to match the live PVC (#336)
Reflect the live 20Gi->40Gi expansion in the manifest as source of truth.
volumeClaimTemplates is immutable on an existing StatefulSet, so landing
this needs a one-time `kubectl delete statefulset gitea -n apps
--cascade=orphan` (pod + PVC survive) for Flux to re-create the STS from
the updated template. Documented inline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:01:29 +02:00
claude-botandClaude Opus 4.8 09945402f3 fix(ci): only rebuild the image whose source changed (#336)
ci/woodpecker/push/build-ci-images Pipeline was successful
build-ci-images had a pipeline-level path trigger but no per-step
filters, so any push touching ci/images/** or secret-rotator/** rebuilt
ALL five images. The recent burst of secret-rotator commits therefore
needlessly rebuilt the four ~1.3G ci-* images each time, re-pushing and
overwriting :latest and orphaning ~5G of registry layers per run — a
primary source of the orphaned-blob pileup that filled data-gitea-0.

Add per-step when:path so each build runs only when its own source (or
the pipeline file) changed, or on a manual run. A secret-rotator change
now rebuilds only secret-rotator.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:25:30 +02:00
claude-botandClaude Opus 4.8 e7e22cdd8e fix(gitea): run package-registry GC every 6h to stop orphan blobs filling the volume (#336)
data-gitea-0 hit 100% and gitea started failing writes ("database or
disk is full"). Root cause was NOT tag accumulation — live images are
only 8.7G and well-deduplicated. It was 10.4G (54% of the volume) of
ORPHANED, unreferenced blobs: every time an image tag is overwritten
(the ci-* images and shelly-manager rebuild frequently, shelly-manager
on both cron and push), the previous multi-GB layer set is orphaned.
Gitea's default cleanup_packages GC runs only @midnight with
OLDER_THAN=24h, so a day's worth of orphans piled up faster than it
collected them and filled the disk. A manual GC run reclaimed nothing
because every orphan was <24h old and thus skipped by the age cutoff.

Set the GC to run @every 6h with OLDER_THAN=4h so orphans are reclaimed
same-day. 4h is well beyond any in-flight push, so a live upload is
never collected. Applying this restarts gitea and its RUN_AT_START GC
reclaims the existing ~10G backlog (usage drops to ~9G).

Emergency relief was already applied out-of-band: the PVC was expanded
20Gi -> 40Gi (ceph-block online resize) to clear the outage without a
risky restart at 0 bytes free; documented on the volumeClaimTemplate.

Supersedes #349 (which targeted tag count, the wrong layer).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:20:36 +02:00
claude-botandClaude Opus 4.8 82f6bc765f fix(secret-rotator): clean up orphaned gitea token on all post-mint failures
ci/woodpecker/push/build-ci-images Pipeline failed
Addresses the review on the gitea-token mode.

Blocking: the cleanup only ran on the Ok(false) gate branch. If token_allows
returned Err (a wrong GITEA_VERIFY_REPO, a 5xx) or the OpenBao store failed
after verification passed, the `?` propagated and the just-minted, write-scoped
PAT was orphaned with no delete and no log — contradicting the stated
guarantee, and unlike ha.rs which logs its orphan. Fixed: verify+store now run
in a closure, and ANY error after the mint deletes the token; if the delete
also fails, a CRITICAL line names the live token so it can be revoked by hand.

Also from the review:
- distinguish 401 (minted token rejected — auth, surfaced as an error) from 403
  (valid token, insufficient scope — a real gate "no") in token_allows, instead
  of collapsing both to Ok(false).
- reject an empty GITEA_SCOPES up front, matching BAO_FIELDS / NB_AUTO_GROUPS.
- add a basic() unit test (gitea.rs had none) pinning the auth-header format;
  crate now has 14 tests.
- tighten the doc comments that overstated the guarantee.

clippy -D warnings clean; 14 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 09:25:20 +02:00
claude-botandClaude Opus 4.8 3862e9a806 feat(secret-rotator): gitea-token mode
ci/woodpecker/push/build-ci-images Pipeline was successful
Rotates a gitea PAT used as a git credential — for Flux's own flux-system
secret. Gitea blocks token-from-token (403), so minting uses BASIC auth with a
dedicated service account's password as the bootstrap, like the Cloudflare and
Proxmox privileged bootstraps.

Make-before-break with a hard safety gate: mint a new token, VERIFY it can both
pull (git-upload-pack) AND push (git-receive-pack, since Flux image-automation
writes back), and only then store it. A token that fails verification is
deleted and the rotation aborts WITHOUT storing — so a bad rotation can never
strand Flux the way a revoked token did. Old rotated tokens (same name prefix)
are reaped after, non-fatally.

The push check is required by default but env-toggleable (GITEA_REQUIRE_PUSH)
for read-only consumers. Scopes, verify-repo, name prefix and field names are
all env-driven.

Validated against the real gitea API on throwaway tokens (zero blast radius,
live flux token untouched):
  - write token -> pull 200 + push 200 -> gate PASS
  - read-only token -> pull 200 + push 403 -> gate REJECTS it (correctly refuses
    to store a token that cannot push)
  - list-by-prefix + delete (reap) works
clippy -D warnings clean; 13 tests pass.

No rotator uses it yet — the ExternalSecret + CronJob follow once CI publishes
the image.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 08:46:07 +02:00
security-bot 8449636d2d chore(image-automation): bump gitea.renner.uno/k.renner/shelly-manager:build.1784697139 gitea.renner.uno/k.renner/vsftpd:build.1784697262
ci/woodpecker/cron/pr-review Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-07-22 06:42:30 +00:00
k.renner 72ec3396d2 Merge pull request 'fix(crowdsec): set tls secretTemplate so cert-manager accepts the certs (#340)' (#345) from claude-bot/crowdsec-cert-secrettemplate into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#345
2026-07-22 06:23:56 +00:00
claude-botandClaude Opus 4.8 3a152ef28f fix(crowdsec): set tls secretTemplate so cert-manager accepts the certs (#340)
The mTLS fix in #343 failed to apply: chart 0.24.0's agent/bouncer/appsec
Certificate templates emit `secretTemplate:` unconditionally while its
child blocks are guarded, so with nothing configured it rendered as null
and cert-manager's webhook rejected it (spec.secretTemplate must be of
type object). The helm upgrade failed and rolled back, so agents stayed
on the fragile register-based path.

Set tls.certManager.secretTemplate.labels so secretTemplate renders as a
valid object. Verified via kubectl apply --dry-run=server against the
live cert-manager webhook: all 4 Certificates now accepted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:46:48 +02:00
k.renner ea2689c6a4 Merge pull request 'fix(crowdsec): mTLS agent auth to end the LAPI-restart auth loop (#340)' (#343) from claude-bot/crowdsec-mtls-340 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/pr-review Pipeline was successful
Reviewed-on: k.renner/homelab#343
2026-07-22 05:26:56 +00:00
k.renner e752495848 Merge pull request 'fix(flux): point GitRepository at the renamed k.renner/homelab URL' (#342) from fix/flux-git-url into master
Reviewed-on: k.renner/homelab#342
2026-07-22 05:23:51 +00:00
claude-botandClaude Opus 4.8 5ff9725d3c fix(crowdsec): mTLS agent auth to end the LAPI-restart auth loop (#340)
The crowdsec-agent DaemonSet CrashLoops whenever the LAPI pod restarts.
Root cause: the LAPI has no persistence (PVCs disabled in 629a303d to
break the Ceph bootstrap circular-dependency), so its sqlite machine-
registration DB is ephemeral. Agents register once (init container) and
cache a password; a wiped LAPI no longer knows it -> 401 "machine not
found" -> CrashLoop -> DaemonSet never Ready -> the crowdsec HelmRelease
upgrade --wait times out -> the whole `infrastructure` Kustomization and
every app that depends on it stays unhealthy.

Enable chart mTLS (tls.enabled + tls.agent.tlsClientAuth, cert-manager
self-signed CA). With client-cert auth the agent no longer runs
`cscli lapi register`; the LAPI auto-recreates each machine record from
the presented cert on connect, so a wiped/restarted LAPI self-heals with
no PVC and no manual re-registration. Verified via `helm template`
against chart 0.24.0: register step gone, 4 certs emitted, zero PVCs.

Supersede the #337 persistence attempt (revert lapi.persistentVolume.
data.enabled to false): it reintroduced the Ceph circular-dependency and
did not heal the agents (fresh volume = empty DB). mTLS makes the
ephemeral DB harmless instead of trying to persist it.

Pin the chart to 0.24.x: the floating ">=0.12.0" silently upgraded to
0.24.0 (changed persistence defaults), part of this incident.

Note in traefik/crowdsec-middleware.yaml that the inert ingress bouncer
now needs https + CA trust before it can be re-enabled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:22:42 +02:00
claude-botandClaude Opus 4.8 8a54323e09 fix(flux): point GitRepository at the renamed k.renner/homelab URL
The repo was renamed gitea_admin/homelab -> k.renner/homelab. Flux's
GitRepository still fetched the old URL, which now 307-redirects, and an
authenticated git fetch does not survive a cross-repo redirect
("authentication required: Verify"). Flux froze at revision 90d7a7c4
(2026-07-21 22:45); nothing merged since was being applied.

Point spec.url at the canonical k.renner/homelab.git so source-controller can
fetch again. Applied live in parallel (kubectl patch) to break the chicken-and-
egg — Flux can't read git to fix the URL that stops it reading git.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:22:02 +02:00
k.renner 25281137cd Merge pull request 'refactor(rotation): port pve-exporter to proxmox-token mode' (#341) from feat/port-proxmox-rotator into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/pr-review Pipeline was successful
Reviewed-on: k.renner/homelab#341
2026-07-22 05:09:39 +00:00
claude-botandClaude Opus 4.8 810b68b11f refactor(rotation): port pve-exporter to proxmox-token mode
The LAST rotator. Replaces the inline python with the secret-rotator binary's
proxmox-token mode. Net -48 lines; inline Python is now fully retired — every
rotator runs the one binary, and the openbao image is left only to the
raft-snapshot job.

Tested live end-to-end via the real binary image — the destructive test that
could not be zero-blast-radius, since it rotates the real monitoring token:
  - job Complete: DELETE+recreated pve-exporter@pve!monitoring (privsep=0),
    OpenBao pve.yml rotated 529d7f66 -> f7ca2141, document well-formed
  - the no-verify TLS path worked against the real in-cluster Service (valid LE
    cert, hostname mismatch) — the piece that could only be compile-proven
  - PVE token recreated privsep=0 (inherits the user's PVEAuditor ACL)
  - ESO synced, Reloader restarted pve-exporter, and it recovered: OpenBao and
    k8s token_values match, the new token authenticates (/version 8.4.18), and
    pve_up=1.0 for the cluster and all three nodes

The brief metrics gap between delete and Reloader-restart is inherent to
Proxmox having no atomic roll (documented in the header); off-hours quarterly
makes it acceptable.

Flux strips the old python `command` on apply (verified on the earlier ports);
port via Flux, not kubectl apply.

Image digest sha256:97c22bcf…

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 07:06:47 +02:00
k.renner 7faf106550 Merge pull request 'feat(secret-rotator): proxmox-token mode' (#339) from feat/proxmox-token-mode into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#339
2026-07-22 04:56:15 +00:00
claude-botandClaude Opus 4.8 af1737a657 fix(secret-rotator): correct proxmox TLS rationale + log DELETE failures
ci/woodpecker/push/build-ci-images Pipeline was successful
Addresses the advisory review on the proxmox-token mode.

The code claimed the PVE cert is self-signed. It is not — the in-cluster
Service presents a VALID Let's Encrypt cert (verified live: CN=neptune.renner.uno,
issuer Let's Encrypt). Verification fails only on HOSTNAME mismatch: the job
connects to the pve.apps.svc.cluster.local Service (for 3-node failover) while
the cert is for a node FQDN. Corrected the module header and the insecure_agent
comment to say this accurately, and spelled out the real tradeoff — ureq 3 has
no skip-hostname-only option, so it's disable-verification-with-failover (this,
matching the python's prior CERT_NONE) vs pin-to-one-node-FQDN-with-full-verify
-but-SPOF. Noted the deferred improvement (a custom rustls verifier that
validates the LE chain but skips hostname).

Also from the review:
- log a warning when the DELETE result is an Err, so a re-run needn't
  rediscover why a first attempt behaved oddly (still non-fatal — a leftover
  token makes the POST fail loud).
- documented why the raw URL-path interpolation is safe (fixed operator config,
  never untrusted; percent-encoding @ to %40 would break PVE's userid lookup).

No behaviour change to the rotation itself. clippy -D warnings clean; 13 tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 06:52:37 +02:00
claude-botandClaude Opus 4.8 1324728721 feat(secret-rotator): proxmox-token mode
ci/woodpecker/push/build-ci-images Pipeline was successful
Adds the last mode — the one that will replace the pve-exporter python rotator,
the hardest of the fleet. Proxmox has no in-place token roll, so this DELETEs
and recreates the token with the same id, then patches OpenBao with a rendered
pve.yml document.

Two capabilities new to the binary:
  - no-verify TLS: a dedicated agent with rustls verification disabled, used
    ONLY for the PVE calls. The API is reached via the in-cluster Service
    ClusterIP, whose cert never matches *.svc.cluster.local, so webpki
    verification can't apply. OpenBao still uses the normal agent.
  - a rendered document, not a bare value: pve.yml is emitted byte-identical to
    the python template (2-space indent, trailing newline, verify_ssl:false),
    unit-tested.

privsep=0 on recreate makes the token inherit pve-exporter@pve's own
permissions, so no separate token ACL is re-applied — verified against the live
cluster earlier (the PVEAuditor ACL is on the user, not the token). The auth
credential is the least-privilege secret-rotator@pve!rotate identity, not root.

DELETE ignores its own outcome (a missing token is a no-op; a real auth/perm
problem surfaces at the POST, where it's fatal — a delete that leaves the token
in place makes the POST fail loud with "already exists"). If the recreate fails
after a successful delete, the store never happens and the error is hard: the
consumer is down until re-run. This destructive window is inherent to Proxmox
and unavoidable without changing the token id each rotation.

Validated against real Proxmox 8.4.18 on a throwaway token (zero blast radius,
never touched the real 'monitoring' token): DELETE 200 -> POST privsep=0 form
-> 36-char value, no trailing newline, and the new token authenticates
(/version 8.4.18). render_pve_yml is byte-exact unit-tested. clippy -D warnings
clean; 13 tests pass.

No rotator uses it yet — the manifest port follows once CI publishes the image.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 00:51:30 +02:00
k.renner 90d7a7c41b Merge pull request 'fix: [alert] KubePodCrashLooping (crowdsec) (#337)' (#338) from alert-fix/issue-337 into master
ci/woodpecker/manual/security-scan Pipeline was successful
ci/woodpecker/cron/pr-review Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#338
2026-07-21 22:44:46 +00:00
k.renner 733e645638 Merge pull request 'refactor(rotation): port netbird-ingress to netbird-key mode' (#335) from feat/port-netbird-rotator into master
Reviewed-on: k.renner/homelab#335
2026-07-21 22:43:43 +00:00