Replaces the inline python NetBird rotator with the secret-rotator binary's
netbird-key mode. Net -30 lines; the login+POST+parse python and the
python:3.13-alpine image go away.
Mints a fresh reusable setup key via POST /api/setup-keys and patches it into
OpenBao. Create-new, not roll-in-place: the old key is left to expire, so
there is no window — it stays valid while the netbird-ingress DaemonSet
re-enrolls its pods on the new one. Nothing is deleted (deliberately — a past
manual delete-filter bug wiped unrelated keys).
Tested live end-to-end via the real binary image:
- job Complete: minted 'ingress-rotated', OpenBao rolled 8a0aa8d4 -> 77f1be0c
- NetBird now has 2 'ingress-rotated' keys (new added, old kept — create-new)
- ESO synced; Reloader restarted the DaemonSet; rolling update (maxUnavailable
1) completed 3/3 with 0 errors
- all 3 pods re-enrolled: the new key shows used_times=3, last_used just now;
pods actively handshaking peers (WireGuard/relay connected)
- old key still valid, untouched
Flux server-side apply strips the old python `command` (verified on the other
ports); a manual `kubectl apply` would not, so port via Flux.
Image digest sha256:49bfddd8…
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the mode that will replace the NetBird python rotator. Mints a fresh
reusable setup key via POST /api/setup-keys and patches it into OpenBao.
Create-new, not roll-in-place: the old key is left to expire (keys are
ephemeral with a bounded expiry), so nothing is deleted — no destructive step,
no window. This deliberately does NOT delete old keys; a past manual cleanup
that filtered keys for deletion wiped unrelated ones (a jq precedence bug), and
that risk has no place in an automated rotator.
NetBird auth is `Authorization: Token <pat>` — NOT Bearer. The key/type/
usage_limit semantics (reusable, unlimited) are fixed; name, expiry, ephemeral,
allow_extra_dns_labels and auto_groups come from the environment. The returned
key is used verbatim (serde_json, no trailing newline). assert_fields_exist
guards the target path. An error that echoes the response redacts the `key`
field so a value can never leak into logs.
Validated against the real NetBird API on a throwaway key (created then
deleted, zero blast radius): TLS verifies with the built-in webpki roots (no
no-verify context needed), POST with Token auth returns a 36-char key with no
trailing newline and the expected settings (reusable/ephemeral/usage_limit 0/
1 auto_group). clippy -D warnings clean; 12 tests pass; the in-cluster pieces
are the same code already proven live by the other modes.
No rotator uses it yet — the manifest port follows once CI publishes the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-ci-images.yml was the last consumer of gitea_token. It used the
admin-owned credential purely as a container-registry login, which meant a
credential that could do anything the admin account could was sitting in a
pipeline that only ever needed to push images.
Replace it with ci_images_token: a PAT scoped to `package: read+write` and
nothing else. Verified — 403 on /user/repos, /admin/users and the issues
API; 200 on the /v2/ registry endpoints kaniko authenticates against.
gitea_token is now unreferenced across every pipeline and can be removed
from Woodpecker and rotated.
The token remains OWNED by gitea_admin because Gitea scopes packages to a
user/org namespace and these images live under gitea_admin/*; a different
account cannot publish there. Moving them to a dedicated org is a
38-reference migration across pipelines and Flux-deployed CronJobs and is
tracked in #330.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the admin-owned gitea_token from every pipeline that runs an LLM
agent, and stops issue-fix borrowing the security-review pipeline's
identity. Each pipeline now holds a credential that can do only what that
pipeline needs.
alert-monitor -> alert_monitor_bot_token (write:issue)
issue-fix -> issue_fix_bot_token (write:issue, write:repository)
update-audit -> update_audit_bot_token (write:issue)
The accounts are write collaborators, but the PAT scope is the effective
cap — verified empirically: the write:issue-only tokens return 200 on the
issues API and 403 on /user/repos and /admin/users.
Also retires two stale rationales in the comments (the gitea_token
workaround and the security_bot_token 403 that motivated it), and changes
issue-fix's git identity from security-bot@renner.uno to
issue-fix-bot@renner.uno so alert fixes no longer appear as
security-review output.
build-ci-images.yml still uses gitea_token for registry/release writes and
needs a write:package token before gitea_token can be retired and rotated;
tracked in #330.
Refs #330. Closes#197.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the two identical Cloudflare python rotators with the secret-rotator
binary's cloudflare-roll mode (added in the prior PR). Net -65 lines; the
duplicated OpenBao-login + PUT-and-parse python and the python:3.13-alpine
runtime image go away.
Both roll their token value in place (PUT /user/tokens/{id}/value) and patch
OpenBao. They differ only in two env values, both parameterised:
cloudflare BOOTSTRAP_TOKENID_FIELD=token-id -> infrastructure/cloudflare
external-dns BOOTSTRAP_TOKENID_FIELD=external-dns-token-id -> apps/external-dns
The admin token comes from the same shared bootstrap, so it stays in one place.
Behaviour preserved: roll-in-place keeps the token id/scope (no window), the
API value is stored verbatim (no trailing newline), and assert_fields_exist
guards the target path. cert-manager needs no reload (reads fresh per DNS-01
challenge); external-dns keeps its Reloader annotation (startup env var).
Tested: the cert-manager rotator was rolled live end-to-end via the real
binary image — OpenBao rolled (d6a4d73b -> 9bd47674), new token verifies active
with the SAME id af708481 (roll-in-place), no trailing newline, ESO synced,
27/27 certs still Ready. external-dns is the identical code path (only the
token-id field + target path differ) and was not rolled to avoid a needless
restart; its PUT/parse was additionally proven against the real Cloudflare API
on a throwaway token in the mode PR.
Note: Flux server-side apply strips the old python `command` when it applies
these (verified against the already-ported Class-A CronJobs, which show
command=<none> live). A manual `kubectl apply` would NOT — it merges and keeps
the stale command — so port these via Flux, not kubectl apply.
Image digest sha256:978fe53c…
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the mode that will replace the two identical Cloudflare python rotators
(cert-manager's token and external-dns's). They differ only in which bootstrap
field holds the token id and which OpenBao path receives the value:
cloudflare bootstrap#token-id -> infrastructure/cloudflare#api-token
external-dns bootstrap#external-dns-token-id -> apps/external-dns#api-token
Both read the admin token from the same shared bootstrap, so it lives in one
place; only the token-id pin differs, keeping the two tokens rolling
independently.
Rolls the token value in place via PUT /user/tokens/{id}/value — same id,
scope and policy, new value — so there is no make-before-break and no
destructive window. assert_fields_exist guards the target path (a typo would
strand the real token), and the API value is used verbatim: a stray newline
would surface as "invalid header field value for Authorization" and crash the
consumer (the bug that hit external-dns once), so the string comes straight
from serde_json with nothing appended.
Handles result as either a bare string or {value: ...} so a future API shape
change fails loud rather than storing "null".
Validated against the real Cloudflare API on a throwaway token (zero blast
radius): PUT with empty body returns success + a bare-string value, new != old
with no trailing newline, new verifies active while the old immediately reads
"Invalid API Token" (confirming roll-in-place has no window). clippy
-D warnings clean; 12 tests pass.
No rotator uses it yet — the manifest port follows once CI publishes the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both agent steps hold Gitea write credentials in their environment
(issue-fix holds a push token too), but neither passed --settings, so
their Read tool was unrestricted — unlike update-audit, which got a
Read sandbox for exactly this reason (gitea issue #197).
That left a complete exfil path: Read /proc/self/environ yields the
token, Write puts it in the agent's output file, and the pipeline posts
that output verbatim as a Gitea issue body / PR body on a PUBLIC repo.
Add the same deny-list used by update-audit (/proc, /root, /home, /etc,
.git) to both agents and wire --settings into each invocation. The repo
checkout, .ci-alert-monitor/ and /tmp/alert-fix/ stay readable, which is
all either agent needs.
This is defense-in-depth only. The root cause — CI pipelines holding an
admin-owned token at all — is tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the inline `head -c N /dev/urandom | base64 | tr -d '\n'` +
`bao kv patch` shell in each of paperless, woodpecker, shelly-manager,
authentik and crowdsec-bouncer-key with the secret-rotator binary's `random`
mode. Net -49 lines; the OpenBao login boilerplate (duplicated in all five)
and the busybox/openbao runtime image go away.
Behaviour preserved exactly:
- byte-compatible output — N raw bytes, STANDARD base64 with padding (RFC
4648 unit-tested); 50 bytes for the Django keys (paperless, authentik),
32 for the rest
- merge-patch, so co-located fields are untouched (WOODPECKER_GITEA_*,
authentik smtp-password)
- shelly-manager's two fields (admin-password, api-key) still land in ONE
patch, no split state
- same k8s-auth roles, SAs, schedule, quarterly timing — the roles already
grant read+patch, which the binary's assert_fields_exist needs
Plus a new safety property the shell lacked: assert_fields_exist refuses to
run if a target field is absent, turning a typo into a loud failure with the
old secret untouched, instead of a silent no-op that adds a stray field.
Image pinned by digest sha256:159e5888… (all three ha-token fixes; the shared
OpenBao code path is the same one `random` uses).
Tested against real OpenBao on a throwaway path+role (zero blast radius, no
live consumer touched):
- happy path: two fields rotated to 44-char values (32-byte base64),
co-located field preserved, multi-field single patch
- fail-safe: a nonexistent target field -> job Failed, clear error, the real
field left unchanged (whole patch aborted, no partial write)
All five pass server-side admission dry-run under the restricted PSA.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New Woodpecker cron pipeline that mirrors the alert-monitor / issue-fix
fleet pattern (cron -> headless `claude` -> Gitea API), but reviews open
pull requests instead of triaging alerts:
- .woodpecker/pr-review.yml: hourly `pr-review` cron. Discovers open PRs
across k.renner/{homelab,shelly-manager,authentik-samba-bridge,
homeassistant}, skips drafts/WIP and PRs already reviewed at their
current head sha (dedup marker), caps at 5/run. For each: clones the PR
head, has a cheap Haiku triage agent pick the model by complexity
(haiku/sonnet/opus; security-sensitive paths force opus), runs the
review with that model, and posts a detailed report as a Gitea PR
review (COMMENT, or REQUEST_CHANGES on a real defect — never APPROVE).
- ci/agents/pr-review/triage-prompt.md: model-selection rubric.
- ci/agents/pr-review/prompt.md: the reviewer's task + report format.
Runs under a dedicated `review-bot` Gitea account (secret
review_bot_token) so reviews are visibly bot-authored and independently
revocable. Static review only — no cluster access; agent tools are
Read/Glob/Grep/Write with no Bash.
Not yet active: needs the review-bot account + token, the review_bot_token
Woodpecker secret, and the `pr-review` cron registered (see file header).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rotates prometheus-token, config-sync-token and shelly-manager-llat via the
secret-rotator binary's ha-token mode (HA mints/revokes LLATs only over its
WebSocket API). All three tested end-to-end against live HA before shipping.
Network: HA is the first IN-CLUSTER rotation target, so it needs BOTH netpol
halves — the openbao default-deny only permits TCP egress to
openbao/kube-dns/apiserver/world (the external-API rotators ride `world`):
- home-assistant-rotator-ingress (home-assistant ns): rotator -> HA :8123
- rotator-home-assistant-egress (openbao ns): the egress complement
Reaches HA on :8123 (HA itself, not the :8124 XFF-anchor proxy) via the
home-assistant-bgp Service ClusterIP; the public name hairpins off the LB.
Both dedicated policies, scoped by the rotator label, not rules on the
namespace-wide default-deny (which would leak to every pod on that port).
Reloader wiring — each token's consumer picks up the new value differently:
- prometheus-token: Prometheus config-reloader auto-reloads; no restart.
- shelly-manager-llat: added to shelly-manager's reload annotation (it reads
the token as a startup env var).
- config-sync-token: the config-sync sidecar reads it at startup and lives in
the HA pod, so the HA Deployment gets a reload annotation. This restarts HA
itself quarterly (04:05, off-hours) — accepted so the full-admin token is
lifetime-bounded like the others.
The binary identifies the outgoing token by its JWT `iss`, not client_name
(the stored names are misleading — prometheus-token is "ServiceMonitor",
config-sync-token was "laptop local"), mints under a unique name (HA rejects a
duplicate), and revokes the old token over a fresh new-token session.
OpenBao policy/role bootstrapped out-of-band (rotator-home-assistant.hcl,
committed for provenance; not Flux-managed).
Tested: prometheus-token and shelly-manager-llat rotated live end-to-end —
OpenBao updated, ESO synced, consumer restarted/reloaded, old token revoked and
gone, new token authenticates (Prometheus scrape 200 with 22k metrics;
shelly-manager 0 HA-auth errors).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Found by running the ha-token rotation end-to-end against live HA. The rotation
was fully correct — new token minted, stored, and the old one revoked — but the
job logged a false "OLD token REMAINS VALID" warning.
Root cause: HA binds a WebSocket session to the refresh token behind its access
token. The rotator authenticated with the OLD token and revoked that same token
over that same session, so HA tore the session down as it processed the delete
— the delete succeeded server-side but the ack could not be read, surfacing as
"connection closed by HA". Verified: after the false-warning run, the old
token's refresh id was already gone from HA.
Fix: after storing the new token, drop the old session and reconnect with the
NEW token to perform the revoke. The session is then not the one being deleted,
so the ack reads cleanly and the report is accurate.
This matters beyond cosmetics: the false warning fired on every successful
rotation, which would train an operator to ignore it — and a genuine revoke
failure would then be indistinguishable noise. With the fix, a warning means a
real un-revoked token.
Verified end-to-end against live HA on a throwaway token: revoke over the new
session returns success=True and the old token is confirmed gone (where the old
flow got "connection closed"). clippy -D warnings clean; 12 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Found by testing ha-token mode against live HA before wiring up the CronJobs.
Home Assistant refuses to mint a second long-lived access token whose
client_name already exists — and make-before-break necessarily mints while the
outgoing token (same base name) is still present. So every real rotation would
fail at the mint step with "Unknown error", exactly as observed:
same-name mint -> FAIL {'code':'unknown_error'}
unique-name mint -> SUCCESS
The earlier ad-hoc test only passed because it minted a NEW name
("Config Sync") rather than rotating an existing one.
Fix: mint under "<base> [<6 hex>]". Correctness does not depend on the name —
the outgoing token is identified by its JWT `iss`, never by name — so a random
tag is free, and it also avoids colliding with an orphan left by a prior failed
run. The failure mode was safe throughout: mint fails before store or revoke,
so the old token stays valid and stored (a no-op, not an outage).
Verified end-to-end against live HA on a throwaway token: unique mint succeeds,
the outgoing token is found by iss, store patches OpenBao, and the old token is
revoked and stops authenticating.
clippy -D warnings clean; 12 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The homelab repo was transferred from owner `gitea_admin` to `k.renner`.
Gitea keeps a 307 redirect for the old path, so git clone/push (which
follow redirects) kept working — but the pipelines' `curl -sf` API reads
do NOT follow redirects, so instead of JSON they received the redirect's
HTML body and the next `jq` died with:
jq: parse error: Invalid numeric literal at line 1, column 3
Under `set -eu` this failed the step before any issue/PR was created,
which is why every alert-monitor / issue-fix / update-audit / security-scan
cron run has been failing and no alert issues were being filed.
- Replace hardcoded `gitea_admin/homelab` -> `k.renner/homelab` in all
API bases, git-push URLs, and cron-add doc comments.
- Add `-L` to the read curls that pipe JSON into jq, so a future rename
degrades gracefully instead of feeding HTML to jq.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses three review findings, all real bugs rather than style:
1. No timeouts anywhere. The Python this replaces passed timeout=30 on every
call; ureq 3's default agent has none and tungstenite has no read timeout,
so a hung endpoint would stall the Job silently until activeDeadlineSeconds
with no useful error — and the cluster demonstrably has hanging paths (the
ingress-LB hairpin). Now: a shared ureq::Agent with timeout_global(30s)
threaded through every OpenBao call, and a 30s read timeout set on the
TcpStream under tungstenite's socket (handling both plain and rustls
streams).
2. RANDOM_BYTES=0 would generate a near-empty secret, store it, and report
success — the same silent-misconfig class assert_fields_exist guards. Now
floored at 16 bytes (128 bits); every real rotator uses 32 or 50.
3. If mint succeeds but the OpenBao store fails, the freshly minted token is
valid for its full lifespan, referenced nowhere, and never auto-revoked
(revocation only targets the OLD stored id). The store error path now logs
the orphan's refresh-token id — recovered from its own JWT — so it can be
revoked by hand, and makes clear the consumer is unaffected because the old
token is still stored and valid.
clippy -D warnings clean; 10 tests pass; scratch image builds (4.92 MB) and
all error paths exit non-zero.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
paperless, authentik, woodpecker, crowdsec-bouncer-key and shelly-manager all
ran the same shell one-liner:
head -c <N> /dev/urandom | base64 | tr -d '\n'
so the port has to be byte-compatible, not merely similar: N raw bytes, the
STANDARD base64 alphabet (+ and /, not URL-safe), WITH = padding, no newline.
Consumers store these verbatim and none filter the character set, so quietly
switching to base64url or dropping padding would change the credential shape
for no reason. The encoder is verified against the RFC 4648 vectors, which are
what coreutils base64 produces, plus a case asserting + and / rather than - and _.
BAO_FIELDS is comma-separated and each field gets its own independent value,
written in ONE merge-patch. That preserves shelly-manager's behaviour, where
admin-password and api-key must land together rather than leaving a window
with one rotated and one stale.
Adds assert_fields_exist as a pre-flight guard. Merge-patch CREATES a field
that isn't present, so a typo in BAO_FIELDS would add `secret-ky`, report
success, and leave the real credential un-rotated — a silent no-op that would
only surface during an incident. Rotation only ever replaces existing fields,
so a missing field is now a hard error that names both what is missing and
what is actually there.
Byte lengths preserved from the originals: 50 for the Django-style keys
(paperless, authentik), 32 for the rest.
Module is `secretgen`, not `gen` — the latter is a reserved keyword in the
2024 edition. 11 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Revoking by client_name would have destroyed live credentials. Checked the
three stored HA tokens against auth/refresh_tokens by decoding each one's JWT
iss claim, and the names are not what they look like:
apps/home-assistant/prometheus-token -> "ServiceMonitor"
apps/home-assistant/config-sync-token -> "laptop local"
apps/home-assistant/shelly-manager-llat-> "Shelly Manager"
Separate, unrelated "Prometheus" and "Config Reconciler" tokens also exist. A
name-matching rotator pointed at prometheus-token would have revoked
"Prometheus" — a live credential belonging to something else — and left the
actual stored token valid. Same for config-sync.
An HA long-lived access token is a JWT whose iss claim IS its own
refresh-token id, so the outgoing token identifies itself exactly. The
rotator now reads that, confirms with HA that the id really is a long-lived
access token, and refuses to proceed if it is not. No name matching anywhere.
The JWT signature is deliberately not verified: this is not authentication,
only reading which id the token claims, and HA remains the authority via the
auth/refresh_tokens confirmation before any revoke.
Adds a small base64url decoder (payloads are unpadded) with unit tests
covering iss extraction, unpadded decoding, non-JWT input, and a JWT with no
iss. 6 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the inline-Python approach before it grows a tenth copy. Today the
fleet carries ~130 lines of Python embedded in YAML strings across 9 manifests,
with auth/kubernetes/login duplicated in all 9 and the call()/merge-patch
helpers in 4 — none of it lintable, testable, or reviewable as code.
The Home Assistant rotator would have been the worst of them: a hand-rolled
WebSocket handshake with frame masking inside a YAML string. HA mints
long-lived access tokens only over its WebSocket API (verified against
2026.7.2 — auth/long_lived_access_token and auth/delete_refresh_token both
exist; the REST API has no equivalent).
Ships as a static musl binary in FROM scratch: 4.85 MB, one file, no libc, no
shell, no package manager. Nothing to patch and nothing for a compromised
rotator to exec. rustls with webpki-roots compiles the trust store into the
binary, so TLS works without a CA bundle on disk — in-cluster traffic is plain
HTTP, but that keeps the door open to take over the Cloudflare and Proxmox
rotators.
ha-token mode is make-before-break, and the ordering is load-bearing: list
existing token ids for the client_name BEFORE minting (so the new token can
never land in the revocation set), mint, write to OpenBao, and only then
revoke. If the OpenBao write fails, the old token is still valid and still
stored, so the run is a no-op rather than an outage.
A failed revoke is deliberately non-fatal but logged loudly: the new token is
already live, and a token meant to be retired that is still valid is a real
finding, not noise.
Verified: static-pie linked and stripped; image contains only the binary
(3.2 MB binary, 4.85 MB image); /bin/sh absent; exits non-zero on usage error;
cargo test passes. The pipeline path trigger now includes secret-rotator/**,
without which source changes would silently not rebuild the image.
No CronJob uses it yet — the Home Assistant rotator needs a network decision
first: the home-assistant namespace is default-deny-ingress, so openbao cannot
reach it, and the public name hairpins off the ingress LB.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous config set IPv6AcceptRA=false on enp1s0.40, which left the
interface with only a link-local address and made edge-1 unreachable over IPv6
on the service VLAN. IPv6 is the primary address family on this network, so
suppressing it to guard against a routing problem in the legacy family was the
wrong trade.
The concern that motivated it was real -- a second default route from the VLAN
40 lease would make return traffic asymmetric across the two VLANs -- but it
only required suppressing the default route and DNS, not addressing itself.
RAs are now accepted with [IPv6AcceptRA] UseGateway=false and UseDNS=false,
mirroring what UseRoutes=false and UseDNS=false already do for DHCPv4. Both
options verified present in systemd 261 (UseGateway added in v250).
Verified after applying to edge-1:
- enp1s0.40 now holds fd40::/64 and 2003:d6:9717:a702::/64 alongside
10.2.40.153/24
- exactly one IPv6 default route, still via enp1s0 (VLAN 30)
- exactly one IPv4 default route, still via 10.2.30.1 on enp1s0
- no DNS servers on the VLAN 40 link
- chronyd and keepalived unaffected
Separately discovered and NOT addressed here: all four NTP servers bind only
0.0.0.0:123, so the service has no IPv6 listener at all -- this predates edge-1
and affects jupiter, saturn and neptune equally. No -4 flag is set and IPv6 is
not disabled in the kernel; the cause is not yet determined.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The VLAN 40 prerequisite is now in place and verified: enp1s0.40 is up with a
DHCP lease of 10.2.40.153/24, and both jupiter (10.2.40.201) and the VIP
(10.2.40.4) answer at ~0.5ms over that interface, confirming layer-2 adjacency
with the existing VRRP members.
NOT APPLIED. The playbook run was blocked pending authorisation, so edge-1
currently has chronyd active and serving but keepalived not installed and the
VIP not held. Applying this flag is what actually joins the VRRP group.
Expected on apply: edge-1 comes up BACKUP at priority 70, below neptune's 80,
so the VIP stays with jupiter. The failure mode to watch for is edge-1 not
seeing the existing advertisements, concluding it is alone, and claiming
10.2.40.4 -- a duplicate holder of the address ~150 clients use. Verify
immediately after applying that edge-1 is BACKUP and does not hold the VIP,
and stop keepalived if it does.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prerequisite for enabling keepalived on edge-1: VRRP needs layer-2 adjacency
with the other NTP members on the service VLAN, and the host has only untagged
VLAN 30. The switch port already permits VLAN 40.
Built to be additive and revertible, because enp1s0 is this headless host's
only path to the network:
- The parent's 20-wired.network is not edited. The VLAN is attached through
a 20-wired.network.d/ drop-in instead, so reverting is deleting three paths.
- The VLAN .network sorts before 20-wired.network and matches explicitly on
Name=. 20-wired.network matches broadly on Type=ether; a VLAN link types as
'vlan' rather than 'ether' (confirmed against networkctl output) so it
should not match anyway, but ordering makes that belt-and-braces.
- UseRoutes, UseDNS, UseNTP and IPv6AcceptRA are all disabled on the VLAN.
Taking a second default route and DNS from the VLAN 40 lease would make
return traffic asymmetric across the two VLANs; untagged VLAN 30 stays the
only default route.
Tasks are tagged vlan40 so they can be applied without running the whole
edge-host role, which would restart the Z-Wave, open3e and alloy services.
Reconfiguring the parent link is what materialises the VLAN child -- a bare
networkctl reload leaves existing links untouched -- so the handler does that
explicitly. That re-runs DHCP on enp1s0 and is the one genuinely risky step;
worth running with console access to hand.
Not yet applied: the apply was blocked pending authorisation, so edge-1 remains
in its original network state with only 20-wired.network and 80-can0.network.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The three PVE hosts currently serve NTP to ~150 LAN clients via chrony with
keepalived holding VIP 10.2.40.4, handed out by UniFi DHCP option 42. Talos
cannot run either daemon, so the planned bare-metal migration removes this
role entirely. edge-1 is brought into the NTP group to take it over.
chrony-ntp-server gains Debian/Arch support. The two distros differ in three
ways that would each have broken on edge-1: config path (/etc/chrony/chrony.conf
vs /etc/chrony.conf), unit name (chrony vs chronyd), and service user (_chrony
vs chrony). Arch also ships no /var/lib/chrony or /var/log/chrony, deferring
them to tmpfiles.d, so the role now creates them explicitly rather than relying
on pacman hook ordering. systemd-timesyncd is stopped where present, since
edge-1 runs it today and it contends with chrony for the clock.
keepalived-ntp is restored -- install-ntp-ha.yml referenced it but the role was
absent from master, so the playbook could not resolve. Recovered from the
homelab-migration-summary worktree with three fixes:
- The health check hardcoded `systemctl is-active chrony`. On Arch that unit
does not exist, so every check would fail, parking the host in permanent
FAULT so it never takes the VIP -- surfacing only at cutover, once the PVE
nodes were already gone. The unit name is now templated.
- Install branches on os_family. Config path and unit name are identical
across distros here, unlike chrony, so nothing else needed changing.
- The sysctl list is now a variable. It unconditionally set
net.ipv4.ip_forward=1, which is correct on the hypervisors but would make
edge-1 a router between IoT VLAN 30 and service VLAN 40 once the tagged
interface exists, bypassing the UDM's inter-VLAN policy. edge_host keeps
only ip_nonlocal_bind; VRRP does not need forwarding.
The two roles have different reach by design. chrony runs everywhere: edge-1
serves from its VLAN 30 address (10.2.30.108) and `allow 10.2.0.0/16` plus
inter-VLAN routing covers every client, so no network reconfiguration is
needed. keepalived is gated on ntp_keepalived_enabled, currently false for
edge-1, because VRRP needs layer-2 adjacency on VLAN 40 and the host has no
enp1s0.40 yet -- the switch port permits the VLAN but /etc/systemd/network/
has no matching .netdev/.network pair.
edge-1's keepalived priority is 70, the lowest, so it stays BACKUP while the
PVE nodes exist and wins the VIP by default once they are gone, with no
config change at cutover.
Verified: playbook syntax-checks; ntp_cluster resolves to the four hosts;
Arch paths confirmed against the package file list. Not yet run against any
host, and runtime var resolution is unverified (needs the vault password).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
external-dns was the last Cloudflare consumer without rotation automation. It
now holds a dedicated single-zone token, so it can roll like the cert-manager
one: PUT /user/tokens/{id}/value keeps scope and policy, changes the value.
Unlike the cert-manager rotator, this needs Reloader — external-dns reads
CF_API_TOKEN as a startup env var. The annotation added in #311 is exercised
by this job and confirmed working: Reloader restarted the Deployment 10s after
the Secret changed, with no manual intervention.
Reuses the existing cloudflare bootstrap so the admin-token lives in one place,
but pins a SEPARATE token-id (external-dns-token-id) so the two tokens roll
independently. Verified: rolling external-dns left the cert-manager token
untouched and all 27 Certificates True.
ExternalSecret refresh 1h -> 10m to match the rest of the rotation fleet.
Tested end-to-end: job completed in 5s, OpenBao patched, ESO synced, Reloader
restarted the pod automatically, external-dns resumed with 0 errors, and the
stored value verifies as the correct active token with no trailing newline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live inspection of the PVE cluster on 2026-07-19 found two defects.
1. No IPv6 address on any external interface, on all three nodes.
/etc/frr/frr.conf enables "ipv6 forwarding" for the OSPFv3 Thunderbolt
mesh that carries corosync, which sets net.ipv6.conf.all.forwarding=1.
The kernel default accept_ra=1 means "accept RAs only while NOT
forwarding", so the nodes silently discarded the Router Advertisements
the gateway sends on VLAN 40. Every external interface was left with
only a link-local address; the sole non-link-local IPv6 anywhere on the
nodes was the corosync mesh /128, which is static in frr.conf and never
depended on RAs.
Verified the gateway is not at fault: RAs arrive on vmbr1.40 from
fe80::265a:4cff:fea6:fd3c every ~10-15s, carrying both the Telekom GUA
prefix and a ULA, and non-forwarding hosts on the LAN autoconfigure
normally. The PVE nodes are the only hosts running FRR.
Fixed with a sysctl drop-in setting accept_ra=2, the value a forwarding
host needs in order to still autoconfigure its own uplink.
2. saturn's vmbr0 is at MTU 1500 while its own uplink enp2s0f0np0 and both
peer nodes' bridges are at 9000. Since the three bridges share one L2
domain, large frames between saturn's guests and the other nodes' guests
blackhole rather than fragmenting. saturn's vmbr0 stanza is simply
missing the mtu line that jupiter and neptune both have.
The two fixes are separated by risk. The sysctl change is non-disruptive
and applies on every run. The MTU change is staged as a pending PVE network
change and applied only when pve_node_network_apply_pending is set, because
applying it reloads networking on the node and briefly interrupts the Talos
k8s VM and its Ceph OSD. The play uses serial: 1 so nodes converge one at a
time.
The MTU task reconstructs the full bridge definition from a pvesh GET before
issuing the PUT. A PUT on /nodes/<node>/network/<iface> replaces rather than
merges, so passing only --mtu would silently drop bridge_ports,
bridge_vlan_aware and bridge_vids, detaching the uplink and flattening every
SDN VNet.
Not applied yet — no run has been made against the cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous comment asserted the annotation "must" sit on the Deployment
rather than the pod template. That is wrong. An A/B test (two identical
Deployments consuming one Secret, differing only in annotation placement)
showed Reloader acting on BOTH within 10s, naming each in its log.
deploymentAnnotations is still used here, but only to match the prevailing
idiom in this repo — not because pod-template annotations are ineffective.
This also clears the suspicion raised against authentik-server,
authentik-worker and crowdsec-lapi: their placement is fine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
external-dns was moved off the shared 4-zone "Caddy" Cloudflare token onto a
dedicated single-zone token (Kubernetes External-DNS: Zone Read + DNS Write on
renner.uno only). Rotating that Secret alone is not enough: external-dns reads
CF_API_TOKEN as an env var at startup, so it kept serving with the old token
until the pod happened to restart. During this cutover that restart had to be
done by hand.
Add the Reloader annotation via deploymentAnnotations so a future rotation
restarts the Deployment automatically.
Placement matters: the annotation goes on the Deployment, not the pod template.
Reloader has only ever been observed acting on workload-level annotations in
this cluster (woodpecker, paperless, netbird-ingress, pve-exporter); the chart
exposes deploymentAnnotations for exactly this.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#306 landed suspended pending a privsep check and a bootstrap credential.
Both are now done and the job is verified, so enable the schedule.
privsep caveat RESOLVED: pve-exporter@pve!monitoring is privsep=0 and its
ACL (/ -> PVEAuditor) is on the USER, not the token. DELETE+re-create with
privsep=0 therefore loses no permission and needs no ACL re-grant.
Bootstrap credential is a dedicated least-privilege PVE identity rather than
root@pam. PVE rejects per-user ACL paths and authorises the token endpoints
via a userid-group check, so the tightest available scope is a group whose
only member is the target user:
role SecretRotator (User.Modify)
user secret-rotator@pve
ACL /access/groups/secret-rotated -> SecretRotator
Verified it can mint/delete pve-exporter tokens, sees no VMs or storage,
and gets 403 minting a root@pam token.
api-url is the in-cluster Service (all three nodes back it). The external
name pve.renner.uno cannot be used from a pod: pods cannot hairpin to their
own cluster's ingress LB, which hangs the job. A node IP or single node
FQDN would be a needless SPOF.
Tested: two full rotations; OpenBao patched, ESO synced, Reloader restarted
the exporter, pve_up returned 1.0 for all three nodes both times.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Staged verification found cert-manager was using an unrelated 4-zone token named 'Caddy' (renner.uno + 3 personal zones, shared with something outside this repo), not the dedicated 'Kubernetes Cert-Manager' token that already existed (renner.uno only). Switched cert-manager to the dedicated token and pinned the rotation token-id to it, so Caddy is never rolled. Net: cert-manager drops write access to 3 personal zones.
Verified end-to-end: job rolls token af708481, OpenBao rotates, rolled token stays active/same scope, ESO syncs, cert-manager writes the DNS-01 TXT record (publicly resolvable), 27/27 prod certs Ready. suspend: false.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
issue-fix-bot
security-bot
claude-botandClaude Opus 4.8
k.renner