Agents outside the pod (Claude Code on the workstation, via
`paperclipai agent local-cli` / agent API keys) only make outbound REST
calls to /api/* with Authorization: Bearer. The Authentik gate on
paperclip.renner.uno rejects those (intercept_header_auth), so they get a
separate machine door, following the wiki-index pattern:
- paperclip-api.renner.uno (no dns.renner.uno/public label: LAN-only)
routes /api to the paperclip Service, bypassing the gate. Paperclip's
bearer agent/board keys are the authentication.
- /api/auth on that host is refused (Traefik ipAllowList matching no
address -> 403), so Better Auth password sign-in/sign-up stays behind
the gate; upstream serves all password login under /api/auth.
- The host is added to PAPERCLIP_ALLOWED_HOSTNAMES (private-hostname
guard would 403 it) and the CNP admits Traefik on 3100.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in this repo checked an alert expression. ci-manifests-test.yml proved
every PrometheusRule assembled into valid YAML and stopped there — no check that
an expression parses, that a label join matches anything, or that a `for` window
behaves the way its comment claims.
On REN-129 the CNPG surge rule was written `for: 2m` and reviewed at that value.
Against Iris's seven measured nights it fires on six of them: two owner emails a
night, 04:16 and 04:22 local, for a WAL backlog that clears itself in under ten
minutes. The arithmetic that caught it was hand-written from scratch twice, in
two runs, and thrown away both times because there was nowhere in the repo to
commit it. It is now a negative control.
Adds:
* scripts/ci/promtool-rules.sh — discovers every PrometheusRule (17 today),
strips `spec:` into a temporary mirrored tree, runs `promtool check rules` on
each, then `promtool test rules` over the committed suites. Discovery floors
on both, as in the sibling steps.
* a `promtool` step in ci-manifests-test.yml, `depends_on: []` like the other
four, on prom/prometheus:v3.15.0. The 3.x pin is load-bearing: the cluster
runs kube-prometheus-stack >= 72.0.0, and alert `for` evaluation is
version-sensitive, so a 2.x promtool would check a different evaluator than
production. The script asserts the major version it is handed.
* kubernetes/apps/monitoring-rules/tests/ — the REN-129 suite (14 cases over
the seven measured nights, an adversarial envelope built from the extremes of
the measured fill/drain ranges, two stall shapes, a threshold boundary, a
failover, a multi-instance case and the epoch-zero guard), its generator, and
six negative controls in mutations.txt.
* a mutation phase. Each control breaks one value the rules depend on and the
suites must fail; a control whose sed matches nothing is a failure, not a
skip.
* scripts/ci/test-promtool-rules.sh, run from ci-scripts-test.yml — 17 cases
asserting the script refuses to report success when it has checked nothing.
The mutation phase is not belt-and-braces, it is the point. Two failure modes
bit while writing this and neither changed promtool's exit code: a `rule_files`
entry that did not resolve, where promtool WARNs, loads zero rules, and passes
every `exp_alerts: []` assertion against an empty rule set; and traces generated
against the wrong clock window, where all seven nights were flat zero. Both were
green. A suite that passes proves the rules match the assertions, not that the
assertions reach the rules.
Production: nothing applied to the cluster. No manifest changed, so the
monitoring-rules kustomization builds identically — its `resources:` list is
explicit, and the new tests/ directory is not in it and has no kustomization.yaml
of its own, so Flux does not see it. Worst case is a red step on PRs touching
kubernetes/**; enable_status_check is false on master, so it cannot block a merge
on its own.
Verified with real promtool 3.15.0 against the real tree: 17 manifests parsed
(56 rules), the suite green, all 6 mutations caught. Run again under BusyBox
1.35.0 sh/awk/sed/find/xargs, the toolchain prom/prometheus actually ships.
Guard tests 17/17 under both bash and BusyBox sh. shellcheck 0.11.0 clean at
every severity. woodpecker-cli 3.18.0 lint clean on all 17 pipeline files.
Not run in Woodpecker itself — this PR is its first real pipeline execution.
Revert: revert this commit. It adds files and one step; nothing else reads them.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Criterion 3 passed: the removal test (#1018, merged 101c11f4) deleted monitor
804092841 and the controller logged `Monitor Removed` at 03:57:58Z. This commit
is the deliberate revert of that test, which Zeus sequenced to run only after
the cap reading below. The restored object is byte-identical at the object level
to the version that provably created 804092841.
It also corrects a wrong claim this file asserted. The header stated that a
monitor-count cap was EXCLUDED as the cause of the `www.renner.uno` 403, on the
reasoning that a cap would arrive as HTTP 200 + `stat: fail`. The removal test
refuted that:
03:26:27-03:56:27 account 10/10, seven consecutive `Status Code: 403`
03:57:58 804092841 deleted, account 9
04:01:27 `Monitor Added: www.renner.uno-uptime-monitor`, id
804092931, account back to 10
One freed slot is the only variable. UptimeRobot returns HTTP 403 when the
account monitor limit is reached, so "account is full" is the first hypothesis
for an IMC 403, not a discarded one. Capacity is now settled on fact for
REN-158; REN-8's 18-monitor page against a cap of 10 stays blocked on it.
Expect this restored create to be refused with 403 until the tier is raised —
`www` took the freed slot. Left in git per the capacity ruling so the retry loop
creates it unaided once capacity exists. `audiobooks.renner.uno` stays covered
by its 60s Gatus check throughout, so there is no coverage gap.
Also recorded: the real delete log string is `Removing monitor … from provider`
/ `Monitor Removed`, not the `Deleting Monitor` this file predicted; and the
status page is in all-monitors mode, proven in both directions (9->10 on create,
10->9 on prune) with no API key, which settles REN-9's monitor-list mode.
Verified: js-yaml loadAll gives 2 EndpointMonitors in `uptime-monitor`; the only
functional changed line is the restored object, every other changed line is a
comment; restored spec compared equal to `f20fe8bf^`. Not applied — Flux applies
from master and I cannot merge my own work.
Revert: revert this commit. Flux prunes the object and, because its monitor does
not exist, nothing in UptimeRobot changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#992's header comment says RecordDuration "IS called, by all six controllers",
established by grepping the tagged upstream source trees. Now that the scrape is
merged the claim is measurable, and five of six emit the family: 170 series from
kustomize (33), source (43), image-reflector (68), helm (25) and image-automation
(1), and zero from notification-controller.
Five is permanent, not today-only. notification-controller's three reconcilers
are registered but never walked -- controller_runtime_reconcile_total for
controller=alert/provider/receiver reads 0 over 56 h of uptime with 2 Alerts and
1 Provider live in the apiserver -- so the family is absent from its /metrics
entirely and creating a Flux Alert or Provider does not produce the sixth series.
Comment-only. Every one of the 17 changed lines is a pure comment line and the
comment-stripped file is byte-identical to master (same sha256), so the object
Flux applies is unchanged and no pod rolls.
Refs REN-127, REN-102, REN-97
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
probes-blackbox.yaml:209 cited apps/gatus/configmap.yaml:176-185 as the
place gatus documents the paperless blind spot. That was true at cb9e73bd,
where the sentence was written. A later +29-line insert into configmap.yaml
pushed the block down; at master those lines are now Web Service 9
(music.renner.uno), which carries no prose at all.
The same 10-line Paperless blind-spot comment now runs 205-214.
Comment-only. No spec, no pod template, no new resource.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reviewed and merged by argus-bot (REN-145). Approved at head 088db00e.
Independently verified: the eight GVKs against the single served CRD version in Flux v2.9.5; every CRD field path the config reads; object census = 149 and the reconcile intervals behind the 2h/1h thresholds; the rendered chart keeps --resources and omits --custom-resource-state-only; the whole rendered ClusterRole uses only list+watch with no wildcard; the real kube-state-metrics v2.20.0 binary accepts the rendered config while a malformed control is rejected at startup; the kustomize CI gate green on the simulated merge result, which CI itself did not test.
Own promtool fixtures: 9 alert tests plus a 13.5-day recording-rule simulation, all passing. Warm-up guard empty at 1201 samples and populated at 1297; the pruning case yields 0 under the shipped comparator against 1.1376 under the withdrawn ratio; tag counting survives a kube-state-metrics pod roll.
The missing staleness alert is an accepted, measured reduction; follow-ups tracked in REN-143.
Revert: git revert -m 1 <this merge commit>. Both Kustomizations are prune: true, so nothing is orphaned.
Reviewed by argus-bot (four-eyes: authored by atlas-bot, merged by argus-bot).
Verified against the PR head 17c1dece:
- helmrelease.yaml diff is comments only; parsed YAML of HEAD^ and HEAD is byte-identical (1683 bytes each). No values change, so no helm upgrade and no cilium agent restart.
- ServiceMonitor selector {managed-by: homelab, name: cilium-agent} matches only the new Service; endpoint port `metrics` exists on it; targetLabels k8s-app=cilium present.
- Service selector k8s-app: cilium, headless, distinct name from the chart's cilium-agent Service - no double-scrape or ownership collision if a chart flag is flipped later.
- Revert is one step: the monitoring-rules Flux Kustomization has prune: true, so git revert removes the Service.
- ci-manifests-test and ci-scripts-test green; advisory review-bot: no blocking issues.
REN-156
Approved by argus-bot at head c1b9b259 after re-review (head moved from dd6720f2).
Ceph capacity gate cleared at 03:49Z: replicapool max_avail 168.8 GiB usable, per-OSD 76.1% -> 78.8% worst case against nearfull 0.85, HEALTH_OK.
Four eyes: authored by atlas-bot, merged by argus-bot.
PVCs do not shrink - manifest reverts with git revert <merge-sha> -m 1, volumes do not.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Approved at 7aff5250 (review 1086). CNPGWalArchiveBacklog for: 30m -> 15m; new CNPGWalArchiveBacklogSurge > 64 for 8m; both exprs wrapped in max by (namespace, pod) so a failover does not restart the for timer.
Verified with promtool 3.5.0: 12-case test rules suite over Iris's seven measured nights plus an adversarial envelope case, all passing, with five negative controls each failing. Revert tested: git revert restores the file byte-for-byte.
Closes REN-129.
Deletes the `audiobooks.renner.uno` EndpointMonitor so that the other half of
REN-7 can be proven: an EndpointMonitor creates an UptimeRobot monitor, and
deleting it removes that monitor again. "A monitor that cannot be removed is a
leak." The create side is already shown — monitor 804092841, created
2026-09-26 03:26:27, controller log `Monitor Added`.
What to observe after this merges, keyless and from either side:
* `https://uptime.renner.uno/api/getMonitorList/UgMXmP1iRV` goes from
`psp.totalMonitors: 10` to `9`, and id 804092841 is absent.
* controller log: `Deleting Monitor: audiobooks.renner.uno-uptime-monitor`.
Precondition, checked before writing this: v2.2.13 carries no finalizer, so a
delete while the controller is down is never reconciled and orphans the monitor
permanently. Pod ingressmonitorcontroller-5d577cc58-pgqwg was Ready=True with
0 restarts. Whoever merges this must re-check that the pod is Ready first.
Also records, in the file, why the second pilot object still has no monitor:
UptimeRobot answers its create with HTTP 403 and IMC retries every 300s
forever. 403 is the non-200 branch of `Add()`, so it is not the provider's
validation path — that returns 200 with `stat: fail` — which rules out the
duplicate-URL, monitor-cap and bad-parameter guesses. Tracked on REN-155; the
object is left in place as the reproducer.
Blast radius: removes one EndpointMonitor from `uptime-monitor` and one
UptimeRobot HTTP check every 300s. No Ingress, workload, ingress controller,
DNS record or RBAC rule is touched, and `www.renner.uno` is unchanged.
Revert: revert this commit. Flux re-applies the object and the controller
re-creates the monitor under the same friendly name — a new id, since the name
and not the id is what binds object to monitor. That round trip is also the
idempotent-reconcile check and is worth doing deliberately.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sizing comment blamed the 10-minute archiving stall on Velero repository
maintenance. REN-136 disproves that over seven nights:
- sustained archiver drain is ~14 segments/min, flat whether Kopia is
running or not, so Kopia does not slow the drain;
- Kopia's nearest batch drifts across 02:03-02:24Z while the burst stays
pinned to 02:13Z, and a cause that drifts 20 minutes cannot produce an
effect fixed to the minute.
The real driver is a nightly producer burst: 56-119 segments queued at
02:13-02:15 UTC on all seven nights, against a steady-state queue of 0.
Comment now states that, and adds REN-136's stalled-night framing (6.36 GiB
free survives one stalled night, ~18.6 GiB carries about four).
Comment text only. `spec.storage.size: 32Gi` is unchanged and still the sole
semantic change on this branch — verified by parsing origin/master and HEAD
with js-yaml: with spec.storage.size masked the two parse to identical
objects.
Raised by Argus in review of #1014 as a non-blocking finding.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reviewed at head 3b96b3ab, test-merged clean into master dd2256ea (tree af52adab).
Verified: one file +73/-0; 0 YAML parse failures repo-wide; jobName blackbox-negative-control unique; all four ports re-derived from the paperless Service/Cluster objects; tcp_connect module declared; paperless-ns.yaml still admits monitoring namespace-wide so the controls read 1 today.
Pages nobody: all 16 PrometheusRules enumerated, the only non-job-pinned probe_success rule (HetznerBackupEndpointDown) is instance-pinned to the Hetzner URL; the chart's up-based TargetDown cannot fire because blackbox answers 200 with probe_success 0 and the 5s module timeout stays inside the 10s scrapeTimeout; all 5 blackbox-overview.json panels are pinned to job=blackbox-http; no Grafana-managed alert references probe_success.
Monitoring-only, no traffic-path change. Revert with: git revert -m 1 <this-sha> on a branch, through a PR.
REN-28 §6.4 gate NOT yet taken - reviewer has no Prometheus credential either. PR 2b must not open until the probe_success == 1 reading exists.
The cilium chart renders templates/cilium-agent/service.yaml only when
prometheus.enabled AND (prometheus.serviceMonitor.enabled OR
prometheus.metricsService). infrastructure/cilium/helmrelease.yaml sets the
first and neither of the second, so no cilium-agent Service existed. The
standalone ServiceMonitor in apps/monitoring-rules selected a Service that was
never there: scrape pool serviceMonitor/kube-system/cilium-agent/0 sat at 0
active / 46 dropped targets and cilium_* had zero series.
Consequence: CiliumBPFMapPressureHigh and CiliumBPFMapFull read
cilium_bpf_map_pressure, so both evaluated an empty vector and could never
fire — the exact failure the helmrelease comment says the metrics were turned
on to prevent. cilium_policy_verdict_total and cilium_drop_count_total were
equally absent, which is why REN-140 needed a purpose-built prober to observe
a network-policy deny.
Declares a sibling metrics Service next to the ServiceMonitor instead of
setting prometheus.metricsService in the HelmRelease. The flag is the
chart-native fix and renders exactly this one object, but it is a values
change: helm-controller would upgrade the CNI release and three-way-merge
against live objects, and the configmap.reloader annotation turns any
resulting cilium-config change into a rolling restart of every agent. A
Service applied by the apps Kustomization cannot do that.
Naming follows authentik-outpost-servicemonitors.yaml: distinct name plus
app.kubernetes.io/managed-by: homelab, with the ServiceMonitor selector pinned
to it, so turning either chart flag on later neither collides on ownership nor
double-scrapes the same endpoints.
The helmrelease.yaml change is comment only — verified by rendering the chart
against the values block before and after: byte-identical, so Flux performs no
helm upgrade.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewed and merged by argus-bot (REN-149). Author atlas-bot; four-eyes split intact.
Verified live: no alerting rule can match job="blackbox-ren28-negative" (24 probe_/up/scrape_ rules enumerated; tcp_connect module timeout 5s < scrapeTimeout 10s keeps up==1); pod IPs 10.244.4.166/10.244.2.220 current; probeSelector strict so the release label is load-bearing and is set; monitoring has no egress enforcement.
Temporary. Delete probes-ren28-negative.yaml and its kustomization line once the REN-28 "after" reading is recorded.
Revert: git revert -m 1 <this sha>
REN-28 item 3a. Narrows the navidrome baseline CNP's whole-monitoring-namespace, every-port ingress admit to the blackbox-exporter pod on :4533 and :8686, and repoints two comments elsewhere in the tree that cited the replaced rule.
Reviewed by argus-bot at head 1c1a637c in a private detached worktree. ci-manifests-test and ci-scripts-test green at that head. Blocking review 1065 (a false ICMP justification) resolved by deletion in 1c1a637c.
Revert: git revert -m 1 <this sha>, through a PR.
ha-pg-1 queues WAL for archiving every night: flat 0 until 02:12Z, peak
02:14-02:15Z, drained by 02:22Z, peaks of 56-119 segments on the seven
nights measured 2026-09-19 to 2026-09-26. Point-in-time recovery for
home-assistant is degraded for the ~10 minutes each burst lasts. Nothing
fires, because CNPGWalArchiveBacklog's `for` was 30m.
The cause is a write burst, not IO contention. Drain was a flat 13.0-14.9
segments/min on all seven nights, including the night whose Kopia
repository-maintenance batch started only after the burst had drained;
that batch drifts across 02:03-02:24Z while the peak stays pinned to
02:15Z. Fill is what varies, 18.7-39.7 segments/min. 02:12Z is 04:12
local, when Home Assistant's recorder purge runs. An earlier revision of
this commit named Velero/Kopia as the cause; the 7-night data retracts
that, and the annotations no longer send anyone to the velero namespace.
Three changes, all alert-rule text. Nothing restarts, no storage and no
traffic path is touched.
- CNPGWalArchiveBacklog `for`: 30m -> 15m. This does not fire on the
nightly burst at either value, and should not: every burst in the week
drained itself, and the longest contiguous excursion above 5 was 10
minutes. 15m is for the archiver that has actually stopped, which is
the failure that fills the PVC and stops Postgres.
- New CNPGWalArchiveBacklogSurge, >64 segments (1 GiB) for 8m, warning.
Covers the shape the 15m rule cannot see: a working archiver outrun by
production. 8m because the measured envelope spends at most 5
consecutive minutes above 64 on any night, so 8m clears the worst
observed night by 3 minutes while still catching 1 GiB that is not
draining -- from 119, a healthy archiver is back under 64 inside 5
minutes.
- Both exprs wrapped in `max by (namespace, pod)`. `role` flips on
failover, a changed label is a new series, and a new series restarts
the `for` timer. On the raw series a backlog that persists through a
failover at minute 11 is silent at 16m and 20m and only pages by 25m.
Verified with promtool 3.6.0 against the measured traces, not applied to
the cluster: `check rules` passes, and unit tests cover the ordinary
night, the worst night, the quiet night, a stalled archiver, and a role
flip mid-backlog. Negative controls confirm the tests have teeth -- at
`for: 2m` the surge rule fires at 02:17Z with $value 99 on the ordinary
night, and without `max by` the role-flip case fails.
Revert: `git revert <this commit>` restores both rules exactly. No state
outside Prometheus's rule evaluation is affected either way.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds one Probe, `blackbox-negative-control-paperless`, with its own jobName
`blackbox-negative-control` and four tcp_connect targets in the `paperless`
namespace. The targets exist to be REFUSED: each must read probe_success == 1
at this commit and 0 once REN-28 PR 2b narrows the whole-namespace `monitoring`
admit in paperless-ns.yaml. That flip is the negative test, and it then stays
on as a standing regression alarm.
This is monitoring-only. No traffic-path change, no restart, no new grant, no
policy edit. blackbox-exporter already runs in `monitoring`, whose baseline CNP
is ingress-only, and default-deny.yaml is scoped `namespace: apps`, so the
paperless ingress policy is the only variable that can move these controls.
Deliberately silent: both blackbox alerts are scoped job="blackbox-http", so a
distinct jobName is outside them and four permanently-failing targets page
nobody after 2b. The inverted alert CNPNegativeControlReachable ships in PR 2b,
not here — between 2a and 2b these controls are supposed to read 1.
Appended at end of file so existing line-number citations (including
paperless-ns.yaml's reference to probes-blackbox.yaml:250) do not shift.
Refs REN-28 §5.6, §6.4. Revert: drop this one Probe document.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `fromEntities: cluster` + `icmps` rule added earlier in this branch was
justified by a false premise. Its comment claimed the new `toPorts` was "the
first layer 4 ingress rule in this namespace" — at f177685d the ingress already
ended in an `icmps:` rule, which is itself a layer 4 rule. So the allow was
neither required by this change nor new with it.
It also did not match its own header ("ICMP from the host and the other nodes"
vs `fromEntities: cluster`), and for pods it was redundant: IPv4 type 3 and
IPv6 types 1 and 2 are already in the namespace-wide ICMP rule below it.
ha-pg-allow-cnpg-operator does not transfer as precedent: it is pod-scoped and
carries no `host`/`remote-node` allow, so `cluster` is the only thing admitting
the host there. Four peer baselines with exactly this shape — monitoring,
home-assistant, ai, external-secrets — carry no entity-scoped ICMP allow, and
monitoring is where blackbox-exporter and Prometheus themselves run.
The branch is now exactly the approved REN-28 §4.1 scope: the monitoring
namespace admit is narrowed to the blackbox-exporter pod on 4533/8686, and
nothing else in navidrome-ns.yaml changes.
Revert: `git revert` this commit restores the allow; reverting the whole PR
restores the namespace-wide monitoring admit.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Two of the four REN-28 §7 negative tests need a prober inside `monitoring`
that is not the workload under test. `pods/exec` in `monitoring` was declined
as a wider grant than the change it verifies, so this uses the prober the
cluster already runs: blackbox-exporter, driven by a Probe object, read back
through the existing read-only REN-36 Prometheus grant.
Adds one Probe, jobName `blackbox-ren28-negative`, module tcp_connect, four
targets that are reachable from `monitoring` TODAY and that the REN-28
paperless PR must close:
10.244.4.166:9187 / 10.244.2.220:9187 pprl-pg metrics port -> §7 test 4
pprl-pg-r.paperless.svc:5432 same, stable name
gotenberg.paperless.svc:3000 another paperless-ns workload -> §7 test 2
This is the "reachable before" half. It can only be taken while the current
policy is still in force, which is why it goes in ahead of the paperless PR.
No alert can match it: every rule in prometheusrule-blackbox.yaml is scoped to
job="blackbox-http", and TargetDown cannot fire because a Probe's scrape target
is blackbox-exporter itself, which answers 200 with probe_success 0 on a failed
probe.
No workload is touched, no restart, no policy change, no secret. Revert is
deleting the file and its kustomization line.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One semantic line: spec.storage.size 20Gi -> 32Gi. The sizing comment above it
is rewritten in the same commit because it justified 20Gi on table growth
("leaves room for autovacuum bloat and index churn"), which is no longer the
reason for the number.
The reason is WAL headroom during an archiving stall. 20Gi left 6.63 GiB free;
when WAL archiving stalled for 10 minutes under Velero repository maintenance
(REN-129) the un-archived backlog hit 1.84 GiB, 28% of that free space. Postgres
cannot recycle a WAL segment until the archiver ships it, so a longer stall
walks the volume toward full and a full PGDATA stops the recorder. 32Gi takes
free space to ~18.6 GiB and the same backlog to 10% of headroom.
Owner approved option A on REN-135 (apply now, do not gate on the capacity
read). Options B (archiver maxParallel test) and C (throttle Velero) were
declined, so ha-pg-objectstore.yaml and Velero are untouched.
Verified by parsing the file before and after: with spec.storage.size masked,
the two documents are byte-identical as parsed objects. instances: 2,
storageClass: ceph-block, all 8 postgresql.parameters and resources are
unchanged.
NOT verified here: `ceph df` for the replicapool pool. This runtime has no
kubectl and no Ceph read path — see the PR body.
Revert: `git revert <merge-sha> -m 1` restores the manifest, but the PVC
expansion itself is NOT reversible; PVCs do not shrink.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Per-object Flux readiness cannot come from the controllers: gotk_reconcile_
condition and gotk_suspend_status are declared by fluxcd/pkg/runtime and
populated by nothing at the pinned tags, so the REN-97 scrape yields 2,273
gotk_* series and none of them are about readiness. This turns on
kube-state-metrics custom-resource-state over eight Flux kinds instead, and
builds the standing not-Ready alerts on it.
The not-Ready half is complete. The Ready-but-frozen ImagePolicy half ships
as recording rules only: no threshold is derivable from the history that
exists today, and the measurements are in the rule file rather than left
implicit.
- kube-state-metrics customResourceState + rbac.extraRules (list, watch only)
- prometheusrule-flux.yaml: FluxImageAutomationNotReady (2h),
FluxResourceNotReady (1h), FluxImagePolicyMetricsMissing (absent, 30m),
and four recording rules for the staleness signal
- image-automation README: the "What watches this" section said both cases
were uncovered
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewed and merged by argus-bot (four-eyes: authored by atlas-bot).
Review 1063 on REN-137. Claims 1-3 re-derived from IMC v2.2.13 source and the
vendored CRD; host choice checked against the live status-page feed (9 baseline
monitors, neither target present). Revert: revert this merge commit; prune=true
plus enableMonitorDeletion=true returns the account to the 9-monitor baseline.
Recorded finding (non-blocking): UpTimeMonitorService.Add() returns nothing and
logs every failure, so a rejected newMonitor call is invisible in Kubernetes and
in Flux. A negative status-page result must not be read as "explicit-list mode"
for REN-9 without a controller-log read.
Implements the owner's decision on REN-52 (2026-09-25, option B), tracked by REN-111: one ClusterRoleBinding, built-in `view` ClusterRole, subject the paperclip/paperclip-prometheus-read ServiceAccount from REN-36.
Reviewed at head 230756c3 in a private worktree. Verified: exactly one object, no Secrets, no write verb, no new ServiceAccount; subject resolves against prometheus-read-rbac.yaml:48-52; name paperclip-cluster-view is unique in the repo. Flux CRD coverage confirmed in git — flux-view-flux-system carries aggregate-to-view: "true" (gotk-components.yaml:240) over all six toolkit groups, so the REN-52 pre-approved extra ClusterRole is correctly not added.
Revertibility confirmed rather than assumed: clusters/homelab/paperclip.yaml:14 is prune: true over ./kubernetes/apps/paperclip and the object has no prune: disabled annotation.
Revert: `git revert <this-merge-sha> -m 1`. Flux deletes the binding within the Kustomization's 10m interval and the token falls back to its REN-36 scope (get services/proxy on two named Services). No workload is touched either way.
Not verified here, owed on REN-111: the post-merge SelfSubjectAccessReview from inside the pod, proving the grant took effect and that view-aggregation actually covers the Flux CRDs in the live cluster.
Reviewed and merged by argus-bot (REN-134). CI green on 97b7f79b: ci-manifests-test, ci-scripts-test, pr-review.
One semantic line: OpenBaoUnavailable for: 1m -> 5m. Verified against the repo that 1m was the tightest probe window in the cluster (next tightest 5m), that probe and rule evaluation both run at 30s, and that severity=critical routes to the email receiver rather than a pager. Revert: git revert 97b7f79b.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
navidrome and lidarr both take identity from X-Authentik-Username
unconditionally (navidrome/deployment.yaml), so any pod that can open a TCP
connection to their listeners can claim any username. The baseline CNP
admitted the WHOLE monitoring namespace on EVERY port, which put every
current and future monitoring workload inside that boundary. Only two flows
actually cross it, and both are blackbox probes.
Exposure delta, monitoring -> navidrome namespace:
before: every pod in `monitoring`, all 4 workloads in `navidrome`, every port
after: blackbox-exporter only, :4533 (navidrome) and :8686 (lidarr)
Both selectors and both ports are measured on the live cluster, not inferred:
* `app.kubernetes.io/name: prometheus-blackbox-exporter` resolves to pod
blackbox-exporter-prometheus-blackbox-exporter-57844f6f8d-6gqz8 in
`monitoring` (Prometheus service-discovery pod labels, read through the
REN-36 read-only proxy grant). Chart prometheus-blackbox-exporter-11.19.1.
* `probe_success` is 1 for both
http://navidrome.navidrome.svc.cluster.local:80/ping and
http://lidarr.navidrome.svc.cluster.local:80/ping. Those Services select
`app.kubernetes.io/name: navidrome` / `lidarr` and would have no endpoints
if the pods did not carry the labels, so the probe path is proven end to
end, prober pod to app pod.
* Service :80 -> containerPort 4533 (deployment.yaml) and 8686
(lidarr.yaml). Cilium matches the POD port after service translation, so
`port: "80"` would read clean in review and break both probes.
ICMP rule added, and it is NOT cosmetic. The toPorts above is the first
layer 4 INGRESS rule in this namespace -- navidrome-ns.yaml is the only CNP
selecting these pods, and its ingress was L3-only until now. Cilium drops
ICMP to an endpoint selected by any layer 4 policy unless the traffic is
CT-related, and the existing ICMP allow matches pods only
(io.kubernetes.pod.namespace Exists), which `host` and `remote-node` do not
carry. The kubelet sources the HTTP probes from the host netns, so its PMTUD
path is host-sourced: without the new rule, Destination Unreachable / Packet
Too Big toward these pods are dropped silently and the symptom is a stalled
transfer, not a policy denial. Same rule, same three types, same reason as
ha-pg-allow-cnpg-operator in home-assistant-ns.yaml. This is an ADDITION to
the approved plan text, flagged here because it is mine, not the plan's.
Two citations elsewhere pointed at the rule being replaced and would have
become false -- the defect REN-28 is about, in reverse. Both now cite what a
rule admits rather than a line number, so they cannot go stale again:
* navidrome/deployment.yaml -- the union list of what really reaches the pod
* monitoring-rules/probes-blackbox.yaml -- the justification for why the
REN-21 probes needed no policy change
NOT touched: `host`/`remote-node` (deleting it CrashLoops both apps -- the
kubelet probes come from there), the in-namespace allow, the Traefik and both
gate-outpost allows, the existing ICMP rules, every egress rule, and
ND_EXTAUTH_TRUSTEDSOURCES (item 2 of REN-28, declined and documented in-tree).
No workload restart: a CNP change is picked up by Cilium in seconds.
Revert: `git revert -m 1 <merge-sha>` on master, through a PR like any other
change. Pure CNP + comment change -- no data migration, no restart, no
rollout; Flux reconciles and Cilium re-applies in seconds. Emergency path if
Flux is impaired:
kubectl apply -f <(git show <merge-sha>^:kubernetes/infrastructure/network-policies/navidrome-ns.yaml)
Detection that a revert is needed, inside one 30s scrape interval:
`probe_success == 0` on either navidrome target (alerting already exists in
prometheusrule-blackbox.yaml), or either pod leaving Ready.
REN-28 item 3a, approved by the owner. Paperless (item 3b) is a separate PR
and does not open until >=24h of green probes here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first two of the survey's 14 page rows, and the first objects in this
repo that cause an UptimeRobot write. Both hosts are ungated, both serve a
real application response rather than a gate, and neither has a monitor in
the account today — so the pilot neither duplicates an existing monitor nor
puts anything on the public page that the owner has not already approved.
Uses `spec.url`, not `spec.urlFrom.ingressRef` as REN-7 was written: an
EndpointMonitor can only resolve an ingressRef in its OWN namespace
(pkg/kube/util/url.go:71), `IngressURLSource` has no namespace field, and
every app Ingress lives elsewhere. Reaching them would mean RBAC and a
widened WATCH_NAMESPACE in each production app namespace. Full reasoning in
the file header.
Deliberately omits `uptimeRobotConfig.statusPages`, so that whether these two
monitors appear on uptime.renner.uno on their own answers REN-9's open
question about the page's monitor-list mode.
Also records two source-verified properties that change how this must be
operated, both in comments: a Ready Kustomization cannot prove the monitor
was created (EndpointMonitorStatus is an empty struct, so kstatus returns
Current unconditionally), and v2.2.13 uses no finalizer, so deleting an
EndpointMonitor while the controller is down orphans its monitor.
Verified: kustomize build over all 68 kustomizations, 68 ok / 0 fail; both
rendered EndpointMonitors validated with kubeconform -strict against the
schema extracted from the vendored CRD, with a negative control confirming
the check rejects a bad enum. Lint/render only — NOT applied; Flux applies
from master and these objects do not exist in the cluster yet.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OpenBaoUnavailable went pending at 01:07 UTC on a single 5s HTTPS probe
timeout while OpenBao was healthy, one evaluation short of paging a
critical. At 1m it was the tightest probe-based window in the cluster by
5x, and isolated single-sample probe blips are ambient on the
blackbox-http fleet at roughly one per two hours (three on three
unrelated targets in six hours).
5m matches WoodpeckerDown in the same group. Nothing real is lost: the
rule's own description says existing Secrets keep working and the damage
surfaces at the next ExternalSecret refresh, and OpenBaoQuorumLost
(for: 2m) still covers the fast-moving failure.
expr and severity are unchanged; this only moves the for: window and
replaces the now-wrong "two consecutive failed probes" comment.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only. The narrowing in 16c0c0a7 is unchanged; this records what the
two expressions do with the shape neither of them mentions — an ExternalSecret
with no `Ready` condition.
Review of REN-106 asked for the third state to be made explicit, on the
strength of a metrics census showing `ai/wiki-index-basic-auth` with only a
`Deleted=False` condition. That series is real; the object behind it is not.
ESO v2.11.0 builds the `Deleted` condition on a throwaway object in the
`IsNotFound` branch of Reconcile, purely to retire the object's metric series,
and never persists it — so the series outlives the object until the ESO pod
restarts. That ExternalSecret was retired from git in 419cbed3, its target
Secret is gone from `ai`, and its sync counter is frozen while its six live
siblings advance. On a live object ESO writes conditions of type `Ready` only.
What that leaves is three shapes, all decided: no conditions yet (in-progress,
bounded by `timeout: 5m`), `Ready=True` (current), `Ready=False` (failed for
the two non-retrying reasons, in-progress otherwise). Flux evaluates
inProgress -> failed -> current and falls back to in-progress, so an explicit
`inProgress` expression would be a no-op on all three; the reasoning is written
down instead of encoded twice.
Also noted: `Ready=True` covers reasons `SecretDeleted` and `SecretMissing`,
where ESO reports healthy with no Secret behind it. Unreachable under this
object's default Retain/Owner policies, and a trap for whoever changes them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewed as argus-bot (review 1056) at head 2821629e. Comment-only, reproduced independently in a private worktree: `git diff -U0` against merge-base 64ee49ab has no added or removed line that is not a whole-line `#` comment, and with comment lines stripped both files are md5-identical to master (crd f753baa9c9684002ad01d309e98755ba, rbac 092ddad9c003209d4bd27721c04048b1). Non-comment bytes identical means the parsed YAML is identical, so the applied objects cannot change: no apply, no RBAC change, no restart of the ingressmonitorcontroller pod.
Provenance re-derived from upstream, not taken from the PR body: 2deeba18 is the v2.2.13 tag commit (2026-05-06), aaffd8a8 is a 2-parent master merge commit (2026-09-25), the two chart CRDs are byte-identical, and the vendored body matches upstream at the tag (md5 f753baa9c9684002ad01d309e98755ba over `apiVersion:` to EOF) — so the new header is a check a human can actually run. All seven cmd/main.go + ingress-wrapper.go + url.go line citations in rbac.yaml verified at the tag.
Render check is CI's ci-manifests-test (green at 2821629e, with ci-scripts-test and pr-review; combined state success) — no kustomize in my sandbox either, so I am not claiming a local render.
Revert: `git revert <this-merge-sha> -m 1`. Nothing in the cluster to unwind.
Closes the two follow-ups recorded on the #995 merge (REN-6).
Reviewed and merged by argus-bot (REN-124). Comments only: js-yaml loadAll base vs head parsed-identical for all three files; gatus data["config.yaml"] byte- and hash-identical at 6572 bytes, so the gatus Deployment does not roll. All 7 repointed ranges resolved against the target files at 317ddbe9. Revert: git revert 317ddbe9.
Both found by Argus in the #995 review and handed to REN-6 as follow-ups.
Comments only — every added and removed line starts with `#`, and both files
are md5-identical to master once comment lines are stripped.
rbac.yaml misstated the failure mode of the omitted pods/services grant. It
claimed a denial "costs a log line, not a reconcile". The function's own error
branches do log and continue, but an RBAC denial never reaches them: the
reconciler holds `mgr.GetClient()` (cmd/main.go:168), so the first Service Get
starts an informer and blocks in cache sync on a `context.TODO()` that nothing
cancels. A forbidden list/watch means the Get never returns and the single
reconcile worker (default `--max-concurrent-reconciles=1`, cmd/main.go:84)
stops entirely, while `/readyz` — `healthz.Ping` — keeps the pod Ready. The
omission is still right (the ingressRef path cannot reach this namespace), but
the grant has to precede an Ingress landing here, not follow the breakage.
crd-endpointmonitors.yaml cited aaffd8a866aa09a77030979d938e766895bde5d4 as the
copy point. That is a master merge commit from 2026-09-25, four months after
the v2.2.13 tag the image is pinned to. Now cites the tag commit
2deeba188fbd2860d0b3a7c3590a954a1a709924 plus the md5 of the vendored body, so
the file can be checked against the version actually deployed. The content was
never wrong: that chart CRD is byte-identical between the tag and the commit
the old header named.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PR #997 inserted 9 comment lines into `paperless-ns.yaml` and 21 into
`navidrome/deployment.yaml`. The citations that #997 itself wrote were
renumbered; the citations elsewhere in the repo pointing *into* those two
files were not. REN-116 (merged as a8c48b54) then added one more line to
`paperless-ns.yaml`, shifting everything below it by one again.
Seven pointers repointed, all verified against this commit's tree:
probes-blackbox.yaml:156 navidrome/deployment.yaml 61-74 -> 80-93
probes-blackbox.yaml:218 paperless-ns.yaml 24-26 -> 34-36
probes-blackbox.yaml:221 paperless/network-policy 8-12 -> 22-26
gatus/configmap.yaml:47 paperless-ns.yaml 24-26 -> 34-36
paperless/network-policy.yaml:15 paperless-ns.yaml 30-32 -> 31-33
paperless/network-policy.yaml:16 paperless-ns.yaml 33-35 -> 34-36
paperless/network-policy.yaml:17 paperless-ns.yaml 36-38 -> 37-39
REN-119 specified 34-36 as `33-35` and listed only the first four rows.
Both were correct at `bebf9aad` but are stale after REN-116's one-line
insert, which is above every allow rule in the file. The last three rows
are the same defect from the same commit; leaving them would put the
file back in review for an identical fix.
`paperless-ns.yaml:9-11` (paperless namespace) is above the insert and is
unchanged — verified, not assumed. `navidrome/lidarr.yaml:114-127`,
`lidarr.yaml:29` and `probes-blackbox.yaml:250` were re-checked and are
still correct.
Comments only, and nothing is applied to the cluster. Verified per file:
`js-yaml` loadAll before and after yields identical structures (3 docs,
1 doc, 2 docs), and stripping comment/blank lines from both versions
gives identical text. `gatus/configmap.yaml:47` is in the header above
`data:` (line 62), outside the `config.yaml: |` block scalar, so
ConfigMap `data["config.yaml"]` stays byte-identical at 6572 bytes and
`configmap.reloader.stakater.com/reload: gatus-config` does not roll the
gatus Deployment. All seven edits substitute a range in place; line
counts are unchanged (312/283/97 before and after), so nothing cascades.
Revert: `git revert <this commit>`. Comments only, so a revert restores
the stale numbers and changes no cluster state.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The proxy-grant paragraph named `prometheus-read-rbac.yaml` and
`helmrelease-prometheus.yaml` by bare filename. Both are now repo-relative
paths, matching how the rest of this page cites manifests, so a 403 at 03:00
does not start with a grep.
Docs-only. No manifest changed. `docs/` is outside every Flux Kustomization
`spec.path` (all of them are under `./kubernetes/`), so Flux renders nothing
from this commit.
Follow-up to Argus's non-blocking note 2 on REN-113/REN-47.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewed-by: argus-bot <svc-argus@renner.uno>
Comments-only. Citation re-verified at head a8c48b54: line 250 of
kubernetes/apps/monitoring-rules/probes-blackbox.yaml is the quoted probe URL,
inside the blackbox-http Probe in namespace monitoring. Both versions of
paperless-ns.yaml parse to identical JSON across all three CiliumNetworkPolicy
documents, so re-applying is a no-op against the running policy.
Revert: git revert -m 1 <this commit>
Reviewed and merged by Argus (REN-105), review 1038 at c40c337e.
Delete-path bound re-derived from v2.2.13: GetByName exact FriendlyName equality against the default template {{.Name}}-{{.Namespace}}; GetAllByName and uptimerobot GetAll have no reachable callers. Ships zero EndpointMonitors, so the first reconcile makes no UptimeRobot call and the 9 existing monitors are untouched.
CI green at c40c337e (ci-manifests-test, ci-scripts-test, pr-review). Nothing applied to a cluster pre-merge; verification was lint/render only.
Open follow-ups for Atlas on REN-6, both comment-only: rbac.yaml misstates the failure mode for the omitted pods/services grant (a denial blocks the cached client in cache sync and wedges the single reconcile worker while the pod stays Ready, rather than costing a log line), and the CRD vendor header cites a post-tag master commit instead of the v2.2.13 tag.
Revert: git revert -m 1 of this merge commit.
`paperless-ns.yaml` cited the blackbox probe target as
`monitoring-rules/probes-blackbox.yaml:250`. From
`kubernetes/infrastructure/network-policies/`, that fragment reads as a
sibling `infrastructure/monitoring-rules/` directory, which does not
exist — `monitoring-rules/` lives only under `kubernetes/apps/`.
Use the full repo-root path, matching the style already used six lines
above for `kubernetes/apps/paperless/network-policy.yaml`. Line 250 and
the probe target itself were correct and are re-verified at this commit:
line 250 is
`- http://paperless.paperless.svc.cluster.local:80/accounts/login/`.
Comments only. Parsing the file before and after with js-yaml yields
identical structures for all three CiliumNetworkPolicy documents, and
stripping comment lines from both versions gives an identical diff. No
`spec` change, no new resource, nothing applied to the cluster.
Follow-up from the REN-110 review of PR #997 (merged as 0995e8d5).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Docs-only. One file, docs/superpowers/specs/2026-09-25-paperclip-design.md, +20/-2.
Closes the four documentation items on the REN-47 review bar (1/14, 5, 7, 15) for the
option-B apiserver-proxy grant, and drops the stale Prometheus-only resourceName text
that REN-95 left behind.
BLAST RADIUS: none. Every Flux Kustomization spec.path in this repo is under
./kubernetes/; docs/ is rendered by nothing. Flux applies nothing from this merge and
no workload restarts. No owner-facing surface changes.
REVERT: git revert %(SHA)s -m 1
One commit, one file, no cluster state to unwind. Nothing to re-apply, no token to
rotate, no pod to restart.
Reviewed by argus-bot (review 1031) against manifests, not against the PR body:
Role is one rule, verbs [get], resources [services/proxy], both resourceNames match
prometheus-read-rbac.yaml on master; chart float >=72.0.0 confirmed in
helmrelease-prometheus.yaml. Authored by atlas-bot, merged by argus-bot.
Implements the owner's decision recorded on REN-52 (2026-09-25, option B),
which was never shipped: one ClusterRoleBinding, roleRef the built-in `view`
ClusterRole, subject the ServiceAccount REN-36 created. No new
ServiceAccount, no new ClusterRole, no write verb, no Secrets.
Kept in its own file rather than appended to prometheus-read-rbac.yaml so it
reverts alone, and because that file's header argues for staying narrow —
which this grant is not. Both of that file's now-false claims are corrected
in place: the token is no longer worth only what its Role grants, and the
`paperclip-prometheus-read` name no longer describes it (REN-52 ruled the
name stays, since `serviceAccountName` is in the pod template and renaming it
would restart the Deployment and evict running agents).
No extra Flux CRD ClusterRole: `flux-view-flux-system` already carries
`rbac.authorization.k8s.io/aggregate-to-view: "true"` with get/list/watch on
`*` across all six Flux API groups, so `view` picks them up by aggregation.
REN-52 pre-approved adding one only if that were absent.
Additive RBAC on an existing ServiceAccount: no Pod restart, no remount, no
traffic path changed. RBAC is evaluated per request, so it takes effect on
the already-mounted token.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stakater IngressMonitorController in its own namespace, with zero
EndpointMonitor resources, so it creates nothing on arrival.
- Vendored EndpointMonitor CRD from chart 2.2.13, image pinned to v2.2.13
and deliberately kept out of Flux image automation.
- Namespaced Role only (no ClusterRole): WATCH_NAMESPACE scopes the
manager's cache to uptime-monitor. Verbs derived from what v2.2.13
actually calls; every omission is justified inline in rbac.yaml.
- The ExternalSecret now renders IMC's config.yaml, because IMC reads its
whole configuration from one Secret key rather than an env var. Config
stays in git; only the key is injected.
- Per-namespace CiliumNetworkPolicy, copied from the netbird-service-sync
neighbour: kubelet probes in, DNS + kube-apiserver + world:443 out.
- The Flux Kustomization keeps the healthCheckExprs gate and the
`wait: true` correction that #993 already merged, byte for byte. What
this commit adds to that file is only what is about REN-6: the
EndpointMonitor-vs-annotation mechanism in the header, and why
Deployment health is not a proxy for a current key now that the
controller shares the Kustomization with the ExternalSecret.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Argus's review bar for the option-B proxy grant (REN-47) asks for three notes on
the hand-over page. None of them was ever written down, and the page had also
gone stale: it still named only the Prometheus resourceName after REN-95 added
Alertmanager.
Added, next to `automountServiceAccountToken: true` where the grant is already
described:
- Both granted resourceNames, not just Prometheus (bar item 1/14).
- The Role scopes the HTTP *method*, not the endpoint — `get` refuses
`POST /-/reload` in the authorizer, but every GET endpoint of both services is
reachable, including `/api/v1/status/config` (bar item 5). Verified live this
run: the six `credentials` fields in that config render as `<secret>`.
- Callers must force GET; Prometheus accepts POST on `/api/v1/query` and that
form 403s under this Role (bar item 7).
- The drift defence: a 403 means check the Service name against `resourceNames`
first, because the chart floats at `>=72.0.0` and a rename breaks this path
silently (bar item 15).
BLAST RADIUS: none. One markdown file under docs/, not rendered by any
Kustomization, so Flux applies nothing and no workload restarts.
REVERT: revert this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a PodMonitor for the six Flux controllers in flux-system. Scrape only —
the not-Ready alert rules that were in the first revision of this branch are
removed, because they cannot work at the versions we run.
Why the rules are gone rather than merely deferred: `gotk_reconcile_condition`
and `gotk_suspend_status` are still declared by fluxcd/pkg/runtime v0.110.3
(runtime/metrics/recorder.go), but at the tags pinned in gotk-components.yaml
(source/kustomize v1.9.5, helm v1.6.4, notification v1.9.4, image-reflector and
image-automation v1.2.5) no controller calls RecordReadiness, RecordCondition
or RecordSuspend — zero call sites across all six tagged source trees, only
RecordDuration. An unpopulated GaugeVec exports no series, so those rules would
have evaluated against nothing for ever and the `absent()` dead-man's-switch
would have mailed admin@renner.uno every 12h from 30m after merge. Flux moved
per-object readiness to kube-state-metrics custom-resource-state
(`gotk_resource_info{ready=...}`), which is REN-102 — a prerequisite for the
alert, not a follow-up to it. Caught in review by argus-bot on #992.
What the scrape still buys on its own: `gotk_reconcile_duration_seconds` per
kind/name/namespace, controller-runtime's reconcile/error/workqueue series,
`up{job="monitoring/flux-controllers"}` under the chart's default TargetDown
rule, and the target REN-102 needs in place anyway.
PodMonitor and not ServiceMonitor is forced: four of the six controllers have
no Service, and the two that do publish the artifact/webhook port, not :8080.
No network policy change is needed and that was checked, not assumed —
gotk-components.yaml ships `allow-scraping` admitting every namespace to :8080
and flux-system has no CiliumNetworkPolicy. Nothing in the Flux install is
touched; Prometheus adds a read-only target.
Verified offline only: `kustomize build kubernetes/apps/monitoring-rules` clean
and the PodMonitor renders; all six Deployments carry both selector labels and
the `http-prom` container port. Not applied to the cluster — no kubeconfig in
this account's grants.
Refs REN-97, REN-102.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The healthCheckExprs block merged in #993 treated any `Ready=False` on the
ExternalSecret as Failed. The `flux-failures` Alert forwards every Kustomization
error event to Alertmanager and Flux health checks have no `for:` window, so a
single unlucky poll mailed the owner about something that was about to fix
itself.
REN-106 asked for `reason == 'SecretSyncedError'`, matching wordpress.yaml.
Checked against the ESO the cluster runs (v2.11.0) that is the wrong reason:
`SecretSyncedError` is the reason ESO sets when it returns an error and keeps
retrying — an OpenBao blip, a token renewal and a not-yet-populated key all
produce it. `SecretOwnedByOther` and `SecretImmutable` are the two reasons ESO
returns nil on, i.e. where it has stopped retrying.
So `failed` now names those two, and `SecretSyncedError` falls through to
in-progress and fails the Kustomization at the existing `timeout: 5m`, which
becomes the missing `for:` window. Detection of a genuinely stuck sync moves
from immediate to ~5m; a transient blip no longer reaches the owner at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comments only. No `spec` change, no pod template change, no rendered
manifest change.
Three comments told the reader the ingress side of paperless and navidrome
is locked to the Authentik gate outposts. It is not. `default-deny-ingress`
for both namespaces uses `endpointSelector: {}`, so it selects the app pods
too, and Cilium unions the allow rules of every CNP selecting an endpoint.
The effective ingress set is therefore the outpost PLUS whole namespaces:
`database` and `monitoring` for paperless, and `monitoring` plus the
lidarr-gate outpost for navidrome. Because both apps trust
X-Authentik-Username unconditionally, any pod in those namespaces can claim
any username.
- kubernetes/apps/paperless/network-policy.yaml: replace "Lock the ingress
side to the gate outpost only" with what is actually enforced, citing the
rules in paperless-ns.yaml.
- kubernetes/infrastructure/network-policies/paperless-ns.yaml: note that
the database and monitoring allows below do broaden the trust boundary in
exactly the way the traefik note above warns about, and why they are kept
(CNPG, and the blackbox probe of the paperless login page).
- kubernetes/apps/navidrome/deployment.yaml: ND_EXTAUTH_TRUSTEDSOURCES is
the whole pod CIDR and is not a narrowing; record why it stays that way
(the outpost pod IP is not stable) and list what the CNP really admits.
Verification: `kubectl kustomize` (v1.31.0 / kustomize v5.4.2) over all 68
kustomization roots in the repo renders byte-identical before and after;
every added and removed line in the diff is a comment line.
Revert: `git revert <this commit>`; restores master 3d6c9940 exactly.
Refs REN-108, REN-28.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Make the lidarr and tika ImagePolicy sort keys a total order, and undo the
automated ls42 -> ls11 downgrade that the defective lidarr key caused.
Reviewed and merged by argus-bot; authored by atlas-bot (four-eyes).
Pod roll of deploy/lidarr in ns navidrome signed off by Zeus on REN-91.
Revert: policy only `git revert e50ccaec`; pin only `git revert 430475a7`;
everything `git revert 430475a7 e50ccaec` (restores master ff4ddaa6 exactly).
Refs REN-91.
Nothing in this cluster reported a failed ExternalSecret sync:
`externalsecret_status_condition` held 0 series, ESO's metrics port was never
scraped, and monitoring-rules had no ESO alert. A key that silently stops
syncing was invisible to Prometheus and to Alertmanager.
This is the scrape only. The alert rule is deliberately left out: its `for:`
window has to be tuned against real data, and that data does not exist until
this has been live for a while. Shipping a guessed window to the owner's
channel is the failure mode REN-100 warned about.
A PodMonitor rather than the chart's `serviceMonitor.enabled`, for two reasons
checked against chart 2.11.0:
* There is no metrics Service to select. `metrics.service.enabled` defaults
to false and the HelmRelease overrides nothing, so templates/service.yaml
renders nothing for the controller. A hand-written ServiceMonitor would
match no Service.
* `serviceMonitor.enabled: true` would work but triggers a Helm upgrade of
ESO, re-rendering its Deployment and re-applying its CRDs
(`crds: CreateReplace`). ESO is what all 54 ExternalSecret manifests in
this repo depend on. A PodMonitor gets the same series and touches the
release not at all.
Unlike the Flux PodMonitor, this needs a network policy change: the cluster is
default-deny and external-secrets-ns.yaml admitted only intra-namespace
traffic, host/remote-node and ICMP. Without the `monitoring` grant the scrape
is dropped and the metric stays absent, so the PodMonitor alone would have been
decoration. The grant is port-restricted to 8080, tighter than the equivalent
rule in velero-ns.yaml, and is additive — it removes nothing, so ESO's existing
reachability to OpenBao and the apiserver is unchanged.
Revert: `git revert <sha>`. Removes the PodMonitor and the ingress grant
together, returning to the current no-ESO-metrics state. No ESO object, no
ExternalSecret and no Secret is touched in either direction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment in this file claimed `wait: true` was how a key that failed to
sync became visible. It is not. Flux ships no status reader for
`ExternalSecret`, so kstatus falls back to its generic rule, which inspects
only `status.observedGeneration` and conditions of type `Reconciling` /
`Stalled`. ESO sets none of those — a failed sync is `Ready=False,
reason=SecretSyncedError` — so kstatus returns Current and the Kustomization
goes Ready with no Secret behind it.
Corroborated in-repo rather than from docs: `llama-server` is also `wait: true`
and carries `ai/wiki-embed-basic-auth`, which sat in ESO retry backoff through
~10 failures on 2026-09-20 and needed a manual `force-sync` while its
Kustomization stayed Ready.
Adds a `healthCheckExprs` entry for `external-secrets.io/v1 ExternalSecret`,
the same mechanism already used here for CNPG `Cluster`, MariaDB and the Ceph
CRs, and corrects the comment.
Both expressions are guarded with `has(status.conditions)`, copying
wordpress.yaml. Unguarded, CEL errors rather than evaluating while `status` is
still empty — the state a freshly applied ExternalSecret is in, which is
precisely the case this check exists for.
Blast radius is this Kustomization alone: nothing `dependsOn` uptime-monitor,
and its only object today is the ExternalSecret itself. No existing workload,
ingress controller, DNS record or other Kustomization is touched.
Revert: `git revert <sha>`, or drop the `healthCheckExprs` block. Reverting
restores the previous behaviour (Ready regardless of sync state) and cannot
affect any other Kustomization.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Appends kube-prometheus-stack-alertmanager:9093 to the resourceNames of
Role paperclip-prometheus-read in namespace monitoring. Additive, get-only,
no pod template change.
Reviewed and merged by argus-bot; authored by atlas-bot (four-eyes).
Negative baseline reproduced live via SelfSubjectAccessReview before merge.
Makes the UptimeRobot API key available in the new `uptime-monitor` namespace via ESO. No controller, no monitors, no workload — the key only. First step of REN-2, closes REN-5.
Reviewed by argus-bot against a fresh clone at 25cabaed, not against the PR body. Verified: remoteRef key is mount-relative (store sets provider.vault.path: homelab; `grep -rn "key: homelab/"` = 0 hits); eso-read.hcl already grants homelab/data/*; allowlist is 20 entries, sorted, one occurrence; uptime-monitor is absent from apps/kustomization.yaml and clusters/homelab has no kustomization.yaml, so no double-application; both dependsOn targets exist in flux-system; no duplicate namespace name across the 31 declared; no key material in the diff.
Blast radius is one leaf Kustomization that nothing dependsOn -- deliberately NOT added to infrastructure/post-crds, where an unpopulated OpenBao path would stall the 19 Kustomizations that dependsOn infrastructure-crds.
Merge commit on purpose so the documented revert stays valid: git revert 25cabaed. prune: true then removes the Kustomization, ExternalSecret, Secret and namespace.
Non-blocking finding handed to REN-6: the comment claiming `wait: true` makes a failed ExternalSecret sync visible is wrong -- kstatus reports an ESO ExternalSecret Current regardless of Ready=False, which is why the `ai` ExternalSecret under the wait:true llama-server KS could sit in backoff through ~10 failures on 2026-09-20. Fix is a healthCheckExprs block, as already used for CNPG and Ceph CRs here.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reverts the image change in b858b4aa. ls11 is 3.1.0.4875 as built on
2025-11-19; ls42 is the same upstream build rebuilt on 2026-09-23. The
31 rebuilds in between are LinuxServer security rebuilds of the base
image, so the cluster is currently running ten months of known-missing
base-image patches.
Kept as its own commit so it can be reverted independently of the policy
change, and done here rather than left to the controller: with the fixed
policy merged, image-automation would compute ls42 and push the same two
lines to master itself within ~30 min. Doing it in the PR makes the pod
roll a reviewed change that lands at a known time instead of an
unattended bot commit -- which is the failure mode this card is about.
Blast radius: one rolling restart of deploy/lidarr in ns navidrome, same
shape as the 60s outage b858b4aa caused. Lidarr is a read-only
missing-album view; nothing else consumes it. No other workload moves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lidarr extracted only <build> from <maj>.<min>.<patch>.<build>-ls<n>, so
all 32 published ls rebuilds of build 4875 shared one sort key and the
winner was an arbitrary tie-break. It fired: b858b4aa auto-committed
ls42 -> ls11 to master and rolled the production pod.
numerical cannot order two fields and alphabetical over a concatenation
mis-sorts on width (ls9 vs ls11), so the key becomes a synthetic semver
<build>.<ls>.0 under a semver policy -- width-safe, and equivalent to
(build, ls) lexicographic. The Lidarr major moves into filterTags
because the range now operates on the synthetic key; a 4.x bump needs a
deliberate edit, per the tight-range rule in README.md.
tika has the same defect: extracting <maj>.<min>.<patch> from a
4-component tag discards the image revision. The README called this
theoretical; it is not -- 2.5.0.1/2.5.0.2, 2.6.0.0/2.6.0.1,
2.7.0.0/2.7.0.1 and 2.9.2.0/2.9.2.1 all collide. Every 3.x release so
far is .0, so the revision is pinned in the filter the way
protonmail-bridge.yaml pins its -1 suffix. No-op today; if upstream
ships a 3.x .1 the policy stops bumping instead of picking wrongly.
Verified against the real published tag lists rather than by inspection:
over 1000 lidarr tags the new filter matches 34, produces 34 distinct
keys, has zero publish-order inversions across the visible history, and
selects 3.1.0.4875-ls42. tika reduces to one tag per key and still
selects 3.3.1.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Appends one `resourceNames` entry to the existing `paperclip-prometheus-read`
Role so the read-only apiserver service-proxy path also covers Alertmanager:
kube-prometheus-stack-alertmanager:9093
This is the widening that the file's own header comment said required a new
card and a new review — REN-94/REN-95 are that card, and Hermes confirmed the
target against the live cluster before this was written.
Nothing else about the grant changes. Same single rule, same `verbs: ["get"]`,
same `resources: ["services/proxy"]`, same ServiceAccount, same RoleBinding.
No new rule, no cluster-scoped kind, no second verb. `get`-only still means the
authorizer refuses POST/PUT to either target, so Alertmanager's silence and
`/-/reload` write paths stay denied for the same structural reason Prometheus's
`/-/quit` does.
No NetworkPolicy edit: `network-policy.yaml` already allows egress to entity
`kube-apiserver` on 6443, and the Alertmanager leg of the proxy hop originates
at the apiserver, not at the paperclip pod.
Unlike REN-36 this touches a Role only, not the pod template, so Flux applies
it without restarting the paperclip Deployment and without evicting any agent.
Comment updates are included because the header asserted "ONE VERB, ONE
RESOURCE NAME" and named only the Prometheus proxy URL; both would have become
false next to a second name.
Revert: revert this one commit. The grant returns to Prometheus-only and no
workload restarts.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Makes the key at OpenBao `infrastructure/uptime-robot`, field `api_key`,
available as a Secret in the new `uptime-monitor` namespace and nowhere
else. No new operator: this uses the External Secrets Operator and the
`openbao` ClusterSecretStore the cluster already runs.
Nothing is monitored by this commit. The namespace holds only the key;
the IngressMonitorController, its ServiceAccount and its RBAC arrive
separately (REN-6).
* `data` + `property` instead of `dataFrom: extract`, so only the one
field lands in the cluster rather than every field at the KV path.
* Adds `uptime-monitor` to the ClusterSecretStore `conditions.namespaces`
allowlist, sorted. Without this ESO reports "not allowed to reference
ClusterSecretStore" and the Secret silently never syncs.
* A DEDICATED Flux Kustomization rather than adding the ExternalSecret to
infrastructure/post-crds. That path runs `wait: true` with no explicit
healthChecks and 19 other Kustomizations dependOn it, so an
ExternalSecret whose OpenBao path is not yet populated would never turn
Ready and would stall most of the cluster. Here the blast radius of a
missing key is this Kustomization alone, which nothing depends on.
* No OpenBao policy change: eso-read already grants `homelab/data/*`.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stops a random critical false page on the storage layer. The rule inferred a replication factor from bytes_used/stored, which for an ec:2+1 pool is coding overhead of exactly 1.5 -- the rule's own threshold -- so pool 15 cephfs-bulk paged for 199 minutes on 2026-09-25 with the cluster HEALTH_OK.
Reviewed by argus-bot against live Prometheus using the expression parsed from e3c5c4c0. Fire-window replay: old expr 231 samples on pool 15, new expr 0. Positive control at threshold <100: old selects pools {1,3,5,6,13,15}, new selects {1,3,5,6,13} -- exactly one pool removed and it is the EC one. pool_id survives the join so the annotations still render.
Merge commit on purpose so the documented revert stays valid: git revert e3c5c4c0.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
CephPoolLostRedundancy infers a replication factor from
ceph_pool_bytes_used/ceph_pool_stored and pages critical below 1.5. For an
erasure-coded pool that ratio is not replication, it is coding overhead:
(k+m)/k. Both EC pools in this cluster are ec:2+1, so their correct,
healthy value is exactly 3/2 = 1.5 — the rule's own threshold.
cephfs-bulk (pool 15) reads 480042086400/320028049408 = 1.5000000384. The
margin is 12 KiB in 480 GB, so the comparison is decided by the last bits
of a float division and lands on either side at random. It paged critical
for 199 minutes on 2026-09-25 (12:37-15:56 UTC, two emails) while the
cluster was HEALTH_OK with zero degraded PGs and every OSD up, then cleared
untouched. migration.rgw.buckets.data (pool 12) is the same ec:2+1 shape.
The rule's premise is also wrong for EC: its description claimed a single
OSD loss destroys data, but 2+1 carries a parity shard and tolerates one
OSD loss, the same as replica-2. Summary and description now say
"replicated" and drop that claim.
Gated on the exporter's own `type` label rather than moving the threshold.
Nudging 1.5 to 1.49 would keep a knife-edge and would also narrow the real
replica-2 -> replica-1 signal (1.0) this rule exists to catch.
NO COVERAGE GAP: a genuine EC redundancy loss raises POOL_NO_REDUNDANCY,
which is a live series in ceph_health_detail and is already deliberately in
scope for CephHealthWarn. That reads Ceph's own verdict instead of
inferring one from bytes, which is the right instrument — an EC pool that
loses a shard does not move its stored/used ratio at all.
VERIFIED against live Prometheus (read-only apiserver proxy, REN-36), using
the expression as serialized from this file:
* 7d replay, old expr: pool 15 matches 49 samples (defect reproduced)
* 7d replay, new expr: 0 matches
* fire-window replay (12:00-16:30 UTC): old matches pool 15 at 231/271
steps, new matches nothing
* positive control, new expr with threshold raised 1.5->3.5: selects
replicated pools 1,3,5,6,7,13,14 and excludes both EC pools (12,15) —
the filter removes exactly the EC pools and nothing else
* evaluated cleanly across 17 distinct mgr instances over 30d, so the
on(pool_id) join does not break on mgr failover
BLAST RADIUS: none to workloads. This edits one PrometheusRule; Prometheus
hot-reloads rule files and no pod restarts. Worst case is alerting scope on
one rule. Nothing writes to the cluster, no Ceph object is touched.
REVERT: revert this commit. One file, no state left behind.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
REN-86 asked for one field: `- name: monitoring` on the `paperclip`
Kustomization's dependsOn, because apps/paperclip/prometheus-read-rbac.yaml
(PR #984) puts a Role and a RoleBinding in namespace `monitoring` and a
from-scratch bootstrap was expected to fail applying them.
The premise does not hold, so the field is not added and the reasoning is
written down instead:
- Namespace `monitoring` is not created by the `monitoring` Kustomization. It
is declared in kubernetes/infrastructure/namespaces/namespaces.yaml, applied
by `infrastructure`, which this Kustomization already dependsOn.
- `infrastructure` runs wait: false, so it applies the namespaces on its first
pass and only its healthChecks gate readiness. It cannot report Ready before
the namespace exists, and this Kustomization cannot start before that.
- Role and RoleBinding are core rbac.authorization.k8s.io/v1 kinds, and a
RoleBinding may reference a Role that does not exist yet. Nothing on this
path needs a kube-prometheus-stack CRD or workload, so there is no analogue
of the monitoring-rules/PrometheusRule split.
Adding the edge would also couple the Kustomization that carries the paperclip
pod to `monitoring` readiness, which itself waits on grafana-db,
kube-prometheus-stack, Loki, Alloy and blackbox-exporter for up to 30m.
Comments only: the parsed manifest is identical to master.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Option B on REN-37, owner-approved: reach Prometheus through the apiserver
service proxy under a Role holding one verb, instead of an L4 allow on :9090
that could not tell GET /api/v1/query from the unauthenticated POST /-/quit
that --web.enable-lifecycle puts on the same port.
ServiceAccount paperclip/paperclip-prometheus-read
Role monitoring/paperclip-prometheus-read
get on services/proxy, resourceNames:
["kube-prometheus-stack-prometheus:9090"]
RoleBinding monitoring/paperclip-prometheus-read -> that SA
(subject namespace: paperclip - load-bearing, cross-ns)
paperclip.yaml serviceAccountName + automountServiceAccountToken: true
Authored by svc-atlas, reviewed and merged by argus-bot: author is not the
merger. ci-manifests-test green on 35bcdf10 (kustomize build over the whole
tree). Reviewed against master 3a457bd6 in a fresh clone.
BLAST RADIUS: applying this restarts the paperclip pod (strategy: Recreate)
and every agent running in it. Nothing in production monitoring restarts -
Prometheus, Alertmanager, Loki and the ingress controllers are untouched.
The paperclip Flux Kustomization has wait: true, so a pod that does not come
back shows as a failed reconcile rather than silently.
REVERT - one step:
git revert -m 1 <the sha of this merge commit>
That drops serviceAccountName, restores automountServiceAccountToken: false
and deletes prometheus-read-rbac.yaml; Flux prunes the ServiceAccount, Role
and RoleBinding on the next reconcile (paperclip Kustomization, prune: true).
It restarts the paperclip pod a second time, for the same reason the apply
does.
AGENT-FREE ESCAPE HATCH - use this if the pod does not come back, because
then there is no agent left to run the revert above. Flux re-applies the same
spec every 10m and overwrites a bare rollout undo; the suspend is what makes
the undo stick:
flux -n flux-system suspend kustomization paperclip
kubectl -n paperclip rollout undo deployment/paperclip
# then land the git revert above, and afterwards:
flux -n flux-system resume kustomization paperclip
Names as they are in this repo: `paperclip` is both the Flux Kustomization in
flux-system (clusters/homelab/paperclip.yaml) and the Deployment in namespace
paperclip. No revisionHistoryLimit is set anywhere under kubernetes/, so the
default 10 holds and a previous ReplicaSet exists to undo to. The undo
restores automountServiceAccountToken: false, which closes the read path at
once, before the RBAC objects are pruned. It rolls the pod template back
wholesale, so an image bump that landed after this merge is rolled back with
it - re-apply that after the resume.
Escape hatch text is the merger's, written into this merge commit because
merge condition 2 puts the revert recipe here. The diff is untouched.
Refs: REN-36, REN-37 (option B), REN-47 (review bar), REN-73 (merge card)
Owner picked option B on REN-37: reach Prometheus through the apiserver
service proxy under an RBAC Role holding one verb, instead of an L4 Cilium
allow on :9090.
The port cannot be the boundary. The chart leaves `reloadStrategy` unset, so
the operator defaults it to HTTP and appends `--web.enable-lifecycle`, which
puts `POST /-/quit` and `POST /-/reload` on :9090 with no auth in front of
them. An L4 allow cannot distinguish those from `GET /api/v1/query`. The
apiserver can: it maps HTTP method to RBAC verb, so a Role with only `get`
refuses both in the authorizer before the request reaches Prometheus.
ServiceAccount paperclip/paperclip-prometheus-read
Role monitoring/paperclip-prometheus-read
get on services/proxy, resourceNames:
["kube-prometheus-stack-prometheus:9090"]
RoleBinding monitoring/paperclip-prometheus-read -> the SA above
paperclip.yaml serviceAccountName + automountServiceAccountToken: true
Shape copied from monitoring/ci-diagnostics-rbac.yaml, which has run this
same proxy path for the alert-monitor since 2026-07-16. No network-policy
edit: paperclip-egress already allows kube-apiserver on 6443.
BLAST RADIUS: applying this restarts the paperclip pod, and every agent with
it. Nothing in production monitoring restarts — Prometheus, Alertmanager,
Loki and the ingress controllers are untouched.
REVERT: revert this commit. One commit, four files, no state left behind;
Flux prunes the SA, Role and RoleBinding and rolls the pod back to no
mounted token.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only change to kubernetes/apps/monitoring-rules/probes-blackbox.yaml. Reviewed by argus-bot at head 49bd9271 against base 2b71dda1: 3 documents parse deep-equal, blackbox-http 20 targets before and after, git diff -U0 non-comment non-blank lines = 0, apps/gatus/configmap.yaml untouched. Revert recipe and blast-radius claims re-executed and confirmed.
Comment-only. Rebuilt on top of master (2b71dda1) after #979 merged, so it
adds to that text instead of reverting it.
- EGRESS: names the two ingress-only Cilium policies in the monitoring
namespace and where blackbox-exporter runs; counts re-verified at
2b71dda1 (635 yaml files, 73 Cilium objects, 38 with egress/egressDeny,
0 in monitoring, 1 CCNP scoped to reserved.health, 17 namespaces with
an egress block).
- BLAST RADIUS: what breaks if that dependency is ever broken
(BlackboxProbeFailing x3, BlackboxProbesAbsent stays silent), who
notices (admin@renner.uno, ~10-11 min), and how fast it is undone.
- REVERT: replaced the claim with an executed recipe. All four git revert
candidates CONFLICT at 2b71dda1; the two forward sed deletions were run
and verified (blackbox-http 20 -> 17 and 20 -> 19).
No object changes: 3 documents deep-equal before/after under two
independent parsers, blackbox-http still 20 targets, and `git diff -U0`
has zero non-comment non-blank changed lines. apps/gatus/configmap.yaml
is not touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only change to kubernetes/infrastructure/image-automation/update-automation.yaml.
Reviewed by Argus (svc-argus/argus-bot). Verified comment-only: SHA256 of the comment-stripped file is 4ed11c66f18b8d15 before and after. No object field changes, so Flux's next reconcile is a no-op.
Revert: git revert 4f1c2cf1
Co-Authored-By: Paperclip <noreply@paperclip.ing>
The comment on ImageUpdateAutomation/homelab named gitea_admin as the
owner of the bootstrap token that pushes writeback commits to master.
That is wrong. The pushing account is `flux` (Gitea uid 9): its
contribution history matches every `chore(image-automation)` commit on
master 1:1, while gitea_admin's own contributions stop at 2026-07-21.
Also separates two things the old wording ran together — the commit
author (security-bot@renner.uno, set in this spec) and the pushing
account (the token owner, not set here) — and records that writeback
failure is silent, since no commit is attempted unless a tracked image
moves.
Comment-only. The ImageUpdateAutomation object is byte-identical; no
Kubernetes object changes, nothing for Flux to reconcile.
Refs REN-50
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Comment-only. No Kubernetes object changes on either file: all 3 probe documents deep-equal to master (blackbox-http 20 targets), ConfigMap/gatus-config deep-equal with data["config.yaml"] byte-identical at sha256 8886ef21...6472 (6572 bytes), so Reloader does not roll gatus. Non-comment non-blank changed lines vs merge base 0a82d61f: 0. Reviewed and verified by argus-bot at head 9142b78a; the REVERT recipe was re-tested against its own commit.
The container was OOMKilled (exit 137) at 19:43 under the 4Gi limit.
With server-local runs every concurrent agent's Claude Code process
and MCP helpers live in this container, so the limit caps how many
agents can work at once; exceeding it kills the server and every
in-flight run (process_lost / orphaned_running_run).
Limit 16Gi, request 2Gi (idle server ~1Gi). Nodes have ~93Gi
allocatable each at 12-49% use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment-only, one sentence, one file. Wording is Argus's from the #979
review (REN-38).
The REVERT paragraph ended with "The paperless block below DOES revert
cleanly on its own with `git revert -m 1 0a82d61f`". True at 0a82d61f,
false at bd8690eb, because this same PR edits the paperless block:
from 0a82d61f: revert -m 1 0a82d61f -> CLEAN revert -m 1 16238d39 -> CONFLICT
from bd8690eb: revert -m 1 0a82d61f -> CONFLICT revert -m 1 16238d39 -> CONFLICT
Reproduced here: at bd8690eb the collision is one region, `<<<<<<< HEAD`
at 155 through `>>>>>>>` at 198, the whole paperless block against an
empty side. So the advice failed in exactly the case it was written for
— someone reverting the paperless probe under pressure, four lines below
a warning not to trust the natural resolution on this file. The new text
says what actually happens and how to resolve it, and stays true as the
comment block keeps changing, because it no longer promises clean.
Bar re-verified at this commit, both files against master, js-yaml:
probes-blackbox.yaml docs base=3 head=3, all deep-equal=true
http_2xx/20, http_2xx_wan_edge/3, tcp_connect/2
gatus/configmap.yaml ConfigMap/gatus-config deep-equal=true
data["config.yaml"] sha256 8886ef21…6472 both
sides, identical=true, bytes=6572
git diff -U0 vs master, non-comment non-blank lines: 0
apps/gatus/configmap.yaml is not touched by this commit at all.
Production constraint: nothing applied, restarted or re-rendered. The
public status.renner.uno page does not roll — ConfigMap data is
unchanged, so configmap.reloader.stakater.com/reload has nothing to act
on.
Revert: `git revert <this-sha>` restores the previous sentence and
nothing else.
Refs REN-21, REN-38, REN-44.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only. Every parsed object in both files is byte-identical to
master at 0a82d61f, verified with js-yaml; zero non-comment lines change.
Review follow-ups from REN-22 on the merged #976/#977 probes:
1. The CNP argument was right in its conclusion and wrong in its framing.
cnp-cross-namespace-allow-apps-gatus-to-monitored-services DOES apply to
blackbox-exporter — its Generalization section names it. We are safe
because the allow it demands is already present, not because the note is
about gatus. The old wording is the sentence that makes the next person
skip the check for the next namespace.
2. Records the load-bearing egress fact, which was written down nowhere.
Cilium enforces at both ends; blackbox-exporter can send only because no
policy selecting a monitoring endpoint declares an `egress:` block.
Verified across the repo: 73 Cilium policies, 38 with egress, none in
monitoring, and the one CCNP is scoped to `reserved.health`. 17 other
namespaces already have an egress block, so a routine hardening pass on
monitoring would break these three targets and leave the *.monitoring.svc
ones working.
3. Extends the 200-vs-3xx caveat to all three new targets, not just
paperless. The repo evidence behind every one of them establishes only
`< 400` (kubelet httpGet accepts 200-399; lidarr.yaml:29's `curl -f`
without -L exits 0 on a 302). It proves the paths are ungated, not that
they answer 200. A 3xx gets reverted, not widened.
4. Records the tested revert recipe for #976 in the file. `git revert -m 1
16238d39` conflicts and the natural resolution silently drops paperless
too.
5. Corrects the stale "needs a CiliumNetworkPolicy first" claim in
apps/gatus/configmap.yaml. The correction is written in the manifest
header, ABOVE `data:`, not in place: the stale sentence sits inside the
`config.yaml: |` block scalar, and gatus carries
configmap.reloader.stakater.com/reload, so editing it in place would roll
the gatus Deployment for a comment. ConfigMap data sha256 is unchanged.
Production constraint: nothing is applied, restarted or re-rendered by this
change. No workload, namespace, policy or probe target is touched.
Revert: `git revert -m 1 <merge-sha>`, or drop either file alone with
`git checkout 0a82d61f -- <path>`.
Refs REN-21, REN-22.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reviewed by argus-bot (four-eyes: authored by atlas-bot, merged by argus-bot).
Rebased onto master after #976 by argus-bot; conflict markers only were removed,
the 29 added lines are byte-identical to the original commit bdf55c5b.
blackbox-http 19 -> 20 targets; other two Probe documents unchanged.
Revert: git revert -m 1 <this merge sha> is clean (tested; 20 -> 19, navidrome
and lidarr intact).
Open risk: /accounts/login/ must return 200, not 302 -- module http_2xx passes
only a final 2xx. If it is a 302, revert this PR rather than widening
valid_status_codes. Unverified at merge time: no Prometheus read path. See REN-22.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
paperless.renner.uno is served entirely through the paperless-gate
Authentik outpost. gatus already records the consequence at
apps/gatus/configmap.yaml:176-185: every path including /api/ answers
302 to an unauthenticated caller, so its check of that hostname — and
the UptimeRobot monitor REN-2 adds — is reachability only and stays
green while paperless is dead behind a live gate. This adds the app
health half of the pair, from the in-cluster vantage.
The target is /accounts/login/ because paperless-ngx has no
unauthenticated health endpoint. It is the app's own established
"the webserver answers" signal — the path its startup, liveness and
readiness probes already use — and it renders a Django view rather
than a static file. It does not cover the consumer/celery workers,
and the comment says so.
No CiliumNetworkPolicy change, and no edit to the paperless namespace.
The gatus comment above asserts an in-cluster probe needs one first;
that was written for gatus, which runs in `apps`. paperless-ns.yaml:24-26
already admits the whole monitoring namespace, where blackbox-exporter
runs, and Cilium unions the allow rules of every CNP selecting an
endpoint — the same mechanism apps/paperless/network-policy.yaml:8-12
relies on.
No PrometheusRule change: the target is covered by the existing
BlackboxProbeFailing catch-all and is correctly absent from its
exclusion list, which per prometheusrule-blackbox.yaml:18-33 holds only
instances that have a dedicated rule elsewhere.
Refs REN-21.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Reviewed by argus-bot (four-eyes: authored by atlas-bot, merged by argus-bot).
One file, kubernetes/apps/monitoring-rules/probes-blackbox.yaml, additive only.
blackbox-http 17 -> 19 targets; other two Probe documents unchanged.
Revert: delete the added target lines and comment block (tested clean).
Do NOT use git revert -m 1 on this merge once #977 has landed -- the natural
conflict resolution silently removes the paperless probe too. See REN-22.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
navidrome.renner.uno and lidarr.renner.uno are served entirely through
Authentik outposts. An external probe of either name therefore measures
DNS, TLS, Traefik and the gate outpost — and stays green while the app
behind a live gate is dead. This adds the second signal the
monitoring-vantage-point-rule asks for: an in-cluster probe of the
ClusterIP, which goes red on the app dying regardless of the gate.
Targets are /ping on both, the same path the pods' own kubelet
liveness/readiness probes already use. That it is ungated is not an
assumption: lidarr's reconcile script curls /ping in a loop before it
has read the API key out of config.xml (lidarr.yaml:29).
No CiliumNetworkPolicy change, and no edit to the navidrome namespace at
all. The wiki gotcha
cnp-cross-namespace-allow-apps-gatus-to-monitored-services is about
`apps`, where gatus runs; the influxdb allow-list it describes already
contained `monitoring`. navidrome-ns.yaml:25-27 likewise already admits
the whole monitoring namespace, which is where blackbox-exporter runs.
No PrometheusRule change either: both targets are covered by the
existing BlackboxProbeFailing catch-all and are correctly absent from
its exclusion list, which per prometheusrule-blackbox.yaml:18-33 holds
only instances that have a dedicated rule elsewhere.
Refs REN-21.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Paperclip agents get their own wiki machine identity: OpenBao
apps/ai/wiki-api-keys field `paperclip` (already written, version 5),
mapped to Auth__ApiKeys__paperclip. Reloader rolls the wiki on the new
key.
Agents reach the service in-cluster at wiki.ai.svc.cluster.local:8080,
like the CI agents, so only the app-level API key applies (no Traefik
BasicAuth user needed). The wiki CNP admits the paperclip pod on 8080
and the paperclip egress CNP allows exactly that pod and port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lidarr in the navidrome namespace as a "what is missing" UI: artist
discographies (Wanted -> Missing) and Spotify followed artists / saved
albums / playlists as import lists.
- Mounts navidrome-music-bulk read-write (Lidarr has no read-only root
folder mode); config on ceph-block.
- A reconcile sidecar (same image, curl+jq) reads the API key from
/config/config.xml and on every start sets renameTracks=false,
writeAudioTags=no, no recycle bin, no empty-folder deletes, no
permission changes -- then adds /music as root folder only if absent,
so the automatic library import never runs with file-mutating
settings.
- No download client or indexer.
- AuthenticationMethod=External behind a dedicated lidarr-gate outpost
restricted to a new "Music Librarians" group (k.renner).
- Image automation on the linuxserver build number.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second named key in apps/ai/wiki-api-keys (OpenBao version 4), mapped to
Auth__ApiKeys__paperclip; writes through the API are attributed to it.
Reloader rolls the pod when the Secret changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The cluster-internal name keeps working when runs move to sandbox pods
(option C), where loopback would not reach the server. The name is
added to PAPERCLIP_ALLOWED_HOSTNAMES because the private-hostname guard
otherwise returns 403 for it. Same-namespace traffic is already allowed
by the baseline CNP in both directions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream derives the agents' PAPERCLIP_API_URL from the public URL, so
agent API calls went out through Traefik to the paperclip-gate outpost.
With intercept_header_auth the outpost treats the agent's Authorization
header as an Authentik credential and returns 401; agents could not
create tasks, post cards or update status.
Agents run in the server pod (option B), so PAPERCLIP_API_URL is set to
http://localhost:3100 (upstream keeps an explicit value and passes it to
every run). Loopback passes the private-hostname guard and agent JWTs
still authenticate each call; the gate stays fully enforced for browser
traffic instead of carving out /api/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first admin is claimed, so PAPERCLIP_AUTH_DISABLE_SIGN_UP flips to
true.
Spec: Claude auth is a Paperclip-managed subscription connection created
once via the local sign-in. The server-env CLAUDE_CODE_OAUTH_TOKEN was
never used (stripAiAuthBindings blanks it for managed runs; upstream
accepts no pasted setup-token), so it is dropped from the spec. The
login must run as the node user: exec sessions start as root and
root-owned 0600 credentials are unreadable to the server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Paperclip's private-hostname guard (authenticated+private) returns 403
for any Host but loopback and the public URL. Kubelet probes send the
pod IP, so the pod could never become Ready. Probes now send
Host: localhost.
- Liveness is a TCP check: /api/health runs SELECT 1, so a Postgres
outage would have restarted the server and killed in-flight runs.
- Baseline CNP allowed world on every port; Cilium unions allows across
CNPs, so the app-level :443 restriction was not enforced. world is
now :443-only in the baseline as well.
- automountServiceAccountToken: false. Every agent shares the pod
filesystem; bots bring their own cluster credentials.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Sign-up stays enabled: upstream's first-admin claim requires an
existing signed-in Better Auth account (routes/access.ts
/bootstrap/claim), so disabling it would lock out the claim. A
follow-up flips PAPERCLIP_AUTH_DISABLE_SIGN_UP after the claim.
- Telemetry is off via PAPERCLIP_TELEMETRY_DISABLED=1, which
resolveTelemetryConfig honours; config.json can't carry it alone
because the schema requires a complete file.
- Container starts as root: docker-entrypoint.sh remaps the node user
and chowns /paperclip before dropping privileges with gosu.
- Image pinned to the amd64 manifest digest of nightly; Flux digest
reflection takes over from there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
digestReflectionPolicy: Always requires spec.interval on the ImagePolicy
(admission rejects it otherwise); set to 1h to match the ImageRepository.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The public edge is proxy-1/proxy-2; proxy.renner.uno has not pointed at
netbird.renner.uno since the netcup fleet took over. The proxy that kept
running on the management host has served nothing since the controller
moved every service to TLS passthrough (2026-09-07): passthrough routes
live on the proxy's own :443 listener, while the host's Traefik forwards
HostSNI(*) to its :8443 main listener, which answers "unknown domain".
Its watchdog therefore failed every probe round and ran `docker restart`
every ~3 minutes since 2026-09-07 12:53 (last healthy round). Each
restart registered a fresh proxy peer and pushed a network-map update to
every peer in the mesh — ~10 stale proxy-* identities per ingress peer
and a map update every 1-2 minutes.
- netbird-server: drop the proxy service, its Traefik TCP passthrough,
proxy.env and the proxy vars; `compose up --remove-orphans` removes
the container, and one-off tasks stop/remove both watchdogs and delete
the proxy volumes, config and state.
- install-netbird-server.yml no longer applies netbird-proxy-watchdog;
its pipeline no longer triggers on that role.
- Comments that described the management host as a cluster member now
describe the proxy hosts only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Navidrome (Subsonic API) as the Spotify replacement for phone apps.
- Library PVC on ceph-filesystem-bulk (RWX, 300Gi, mounted read-only);
SQLite data on ceph-block (RWO).
- Web UI behind a dedicated navidrome-gate Authentik proxy outpost;
Navidrome trusts X-Authentik-Username (ExtAuth) and auto-creates the
user on first SSO login (first user = admin).
- /rest and /share routed straight to Navidrome with the identity header
stripped, so Subsonic clients use their Navidrome password.
- Namespace baseline CNP, gate egress CNP, Flux image automation,
Velero daily schedule (09:15), bootstrap wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
immich-server mounts /data from immich-library-bulk (EC 2+1) and
/data/thumbs from immich-thumbs (3x replicated). Data was copied with
rsync -aH from immich-library and verified per folder: file counts and
byte totals match for library, upload, encoded-video, profile, backups
and thumbs.
immich-library stays in pvcs.yaml until a Velero backup of the new
volumes has completed; its removal is a separate change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds immich-library-bulk (ceph-filesystem-bulk, EC 2+1, 300Gi) for
the media root and immich-thumbs (ceph-filesystem, 3x, 30Gi) for
thumbs. Thumbnail reads stay on the replicated pool, where CSI read
affinity serves them from the local OSD; an EC read always
reassembles chunks from two OSDs.
Both PVCs start empty and nothing mounts them yet. The data copy and
the server cutover follow separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ImageRepository and ImagePolicy for gitea.renner.uno/konairius/wiki, same
shape as shelly-manager; the deployment carries the setter marker, so the
existing ImageUpdateAutomation bumps the tag on every main build.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wiki main after PR #12: /facts is one text box; the drafter turns the
statement into a raw source under the signed-in user.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wiki main after PR #11: /facts deposits a raw source under the signed-in
user through the same use case as the API.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Books live on audiobookshelf-books-bulk since #955; nothing mounts
audiobookshelf-books any more. The PVC has prune disabled, so removing
it from git does not delete it; the PVC, its Retain PV and the CephFS
subvolume are deleted by hand after merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wiki main after PR #10: every typed question rides in the single request
that carries the state, halving TypeSafe input tokens per ask.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Points the audiobookshelf Deployment and the libation CronJob at
audiobookshelf-books-bulk (ceph-filesystem-bulk, EC 2+1). The data was
copied with rsync -aH from the old PVC and verified: 574 files,
87315610958 bytes on both sides.
audiobookshelf-books stays in pvcs.yaml (prune disabled) until the
new volume is confirmed in use; its removal is a separate change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Enables StripAudibleBrandAudio (Libation trims the branding using Audible's
own chapter offsets, lossless) and OverwriteExisting, so a re-liberated book
replaces its file instead of landing next to it as 'Title (1)'.
Settings.json on the PVC is seeded once and the entrypoint copies it into
ephemeral /config-internal each run, so the flags are applied there with jq
(already in the image; the entrypoint uses it for Books/InProgress).
Only affects books decrypted from now on; existing files need a re-liberate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds a secondary erasure-coded (2+1, failureDomain host) data pool
'bulk' to the cephfs filesystem and a ceph-filesystem-bulk StorageClass
targeting it. data0 stays the default data pool, as Rook requires for
EC CephFS pools. Bulk media at 1.5x instead of 3x raw overhead.
Behaviour matches the existing migration.rgw.buckets.data EC pool:
Rook sets ec_overwrites, min_size 2 (=k), so IO continues through a
single-node reboot without redundancy for that window.
Also provisions an empty audiobookshelf-books-bulk PVC on the new
class. Deployment and libation CronJob still mount audiobookshelf-books;
the data copy and cutover follow separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The open3e-esp32 gateway on the Vitocal 250-A internal CAN (ECU 0x6e0,
room setpoints) publishes and listens under open3e/int/#. A separate
broker user keeps its command topic out of edge-1's open3e/cmnd listener
and gives the credential its own lifecycle.
Follows the health credential pattern: OpenBao-backed via ExternalSecret,
Class-A rotator CronJob (quarterly, months 3/6/9/12), per-secret policy
and kubernetes auth role, Reloader annotation on the broker.
Out-of-band steps done on 2026-09-24 with an admin token: policy written,
auth role created, path seeded with a 32-char alphanumeric password.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QNSJsCKwMip9y9YEZhDqxw
All credentials reach the service via envFrom, which is read once at pod
start. Reloader watches the four Secrets and restarts the Deployment on a
change, so a corrected or rotated key takes effect without a manual restart.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wiki main at 8fa9f20: POST /api/v1/facts, POST /api/v1/questions with the
Anthropic answer model and the Answers page, POST /api/v1/lab/eval and the
Lab page. The pod restart also picks up the Anthropic key from wiki-anthropic.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wiki PR #7 adds the questions endpoint backed by the Anthropic Messages API.
The key arrives as Wiki__AnswerModels__Models__claude-sonnet__ApiKey via an
ExternalSecret; the secretRef is optional so the pod keeps starting while the
OpenBao key does not exist yet. Namespace egress already admits world, so no
CNP change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The .NET wiki service has parity (same recall on the labelled set), serves
wiki-index.renner.uno and the notes webhook, and the workstation scripts
use it. Removes the app, its Flux Kustomization, image source and build
step, its CNP entries and the OpenBao policy file. The OpenBao role and
the old Gitea webhook are removed live after the merge.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
BasicAuth as before, X-Authentik-Username stripped on that path, Traefik
admitted by the wiki CNP again. The Python service keeps running without
an ingress until the scripts are verified against the new one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Proxy provider, application and outpost blueprint; outpost egress CNP;
ExternalName mirror; the ingress backends the outpost and the BasicAuth
middleware is gone. The ai CNP admits the outpost and Gitea only, so the
X-Authentik-Username header the service trusts from pod addresses cannot
arrive from the LAN.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Gitea's egress to ai was scoped to the wiki-index pod; deliveries to
wiki.ai.svc timed out (hook 17, 2026-09-24).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
PVC, Deployment (build.1790245544), Service, ExternalSecrets for the
shared Gitea token/webhook secret and the API keys, LAN-only ingress
wiki.renner.uno behind BasicAuth, Flux Kustomization, CNP admitting
Traefik and Gitea. No OpenBao access from the pod.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Our hand-written Alertmanager config replaced the chart's default and
dropped two pieces kube-prometheus relies on:
- a route sending InfoInhibitor to the null receiver
- the inhibit rules that let InfoInhibitor (and warning/critical)
suppress severity=info alerts
So both the helper alert and the info alerts it exists to silence were
emailed, daily. InfoInhibitor fired over the last 7 days in crowdsec,
kube-system, woodpecker-ci and without a namespace, driven by
CPUThrottlingHigh (crowdsec-agent, whereabouts-reconciler),
KubeletTooManyPods and KubeQuota* in woodpecker-ci.
Route and the three inhibit rules are copied from kube-prometheus
manifests/alertmanager-secret.yaml. Info alerts now reach email only
while a warning or critical alert fires in the same namespace, which is
the upstream design. amtool check-config passes; amtool config routes
test sends InfoInhibitor to null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Final step of the CVE-2025-30156 remediation: allowedCiphers
[aes256k], which Rook applies as auth_allowed_ciphers. That is step 10
of Ceph's CephX aes256k upgrade procedure and clears
AUTH_INSECURE_KEYS_ALLOWED; per Ceph's health-check reference that also
resolves AUTH_INSECURE_KEYS_CREATABLE.
Precondition from the procedure, checked without printing any key:
AUTH_INSECURE_CLIENT_KEY_TYPE and AUTH_INSECURE_SERVICE_KEY_TYPE are both
clear (ceph health detail lists only KEYS_ALLOWED and KEYS_CREATABLE).
Rook status: daemon keys and CSI/rbdMirrorPeer at generation 2, prior CSI
keys deleted (#934).
Lockout recovery if auth breaks (Rook cephx-key-rotation doc): set
allowedCiphers [aes, aes256k] and daemon.keyType aes, wait for the mons
to carry --mon-auth-emergency-allowed-ciphers, then remove
daemon.keyType.
Closes#527
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
keepPriorKeyCountMax 1 -> 0. Rook deletes the generation-1 CSI keys
(client.csi-*.1, aes), which clears AUTH_INSECURE_CLIENT_KEY_TYPE.
Every PVC has been remounted with the generation-2 aes256k keys: all
three nodes were drained and uncordoned after the rotation, and no
kernel session on any mon uses a .1 entity any more. The remaining .1
sessions are idle librados connections in the CSI nodeplugins' pool;
ceph-csi keys pool entries by user and keyfile, and the CSI secrets now
name the .2 users, so nothing reuses them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Run 10608 aborted before its first edit: the agent calls
~/Notes/bin/wiki-snapshot.sh and git -C ~/Notes rev-parse/status, the
rules carried only /root/Notes. Both spellings are allowed for the vault
scripts and the read-only git subcommands on the vault checkout, config is
denied there too, and the consolidate step derives its --allowedTools
from the settings file instead of a second copy. Verified headless:
rev-parse, status and wiki-delete.sh allowed, config denied.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The last five CephX entities on aes are the four CSI clients and
client.rbd-mirror-peer. The CSI keys waited on node kernel support:
Rook's key-rotation doc asks for kernel 7.0+ for aes256k over kernel
mounts. Talos v1.13.10 (now on all nodes) backports those libceph
patches to 6.18 (siderolabs/pkgs 2da2cdc); CONFIG_CRYPTO_KRB5 and
CONFIG_CRYPTO_KRB5ENC are set on every node. ceph-csi is v3.17.1, the
minimum the doc names; Rook v1.20.7.
csi: KeyGeneration 2, keyType aes256k, keepPriorKeyCountMax 1. Rotated
keys only apply to new mounts; the prior key stays valid for PVCs
mounted before the rotation, per the Rook doc.
rbdMirrorPeer: KeyGeneration 2, aes256k. No CephRBDMirror and no peers
exist, so rotating the peer token affects nothing.
allowedCiphers / mon_auth_allow_insecure_key are NOT changed here: that
is the lockout-capable step and follows only once ceph health shows no
aes entity left.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review on #927: an empty or truncated /tmp/consolidate.json made jq the
failing command instead of the run. Commit subject of push-notes names
consolidate too.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
All three nodes run v1.13.10 (applied by hand, one node at a time:
drain, talosctl upgrade --image, wait for 217/217 active+clean). This
brings the repo in step so a rebuild or a new node installs the same
version.
Why this release: its pkgs include siderolabs/pkgs@2da2cdc, 'backport
aes256k support (Ceph)', the libceph patches from Linux 7.x on top of
6.18.48. Kernel config on every node now has CONFIG_CRYPTO_KRB5=y and
CONFIG_CRYPTO_KRB5ENC=y. That was the blocker for rotating the CSI
CephX keys to aes256k (#527).
Patch release within 1.13; Kubernetes stays v1.36.3. Schematic is
unchanged; factory installer and ISO verified to exist for the tag.
talosctl checksum matches the release's sha256sum.txt and a local
download.
Refs #527
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Draining k8s-3 for the Talos 1.13.10 upgrade stopped on four pods
(wp-pg-1, ha-pg-1, gf-pg-1, gitea-0) whose PDBs allow zero disruptions,
and each needed a manual --disable-eviction.
CNPG: ha-pg, gf-pg, wp-pg and tr-pg go to instances: 2. With a replica,
CNPG switches the primary over ahead of a drain, the same way ak-pg,
im-pg and pprl-pg already do. With one instance it creates a primary PDB
and 'will prevent draining the node where the instance is housed'
(CNPG docs, kubernetes_upgrade). smartShutdownTimeout 10 / stopDelay 60
as on ak-pg, which bound switchover time and the barman-plugin demote
hang (cloudnative-pg#9770). Network policies already allow intra-cluster
replication, operator :8000 and WAL-restore egress in all four
namespaces; nothing references a specific instance name.
Single-replica RWO workloads (gitea, woodpecker-server,
protonmail-bridge, syncthing, influxdb): drop the minAvailable:1 PDBs.
They cannot fail over, and eviction already stops the pod gracefully
with SIGTERM and its grace period; the PDB only forced a manual
--disable-eviction on every drain. influxdb's comes from the chart
(pdb.create, influxdb2 2.1.2 defaults).
The 'single-instance cluster, the archive is the only off-node copy'
comments no longer hold and are removed (im-pg's was already stale).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review on #927: the git -C /root/source/ prefix let config and push
through; rm /root/Notes/wiki/ matched path traversal. Allowed git forms
are now log/show/diff/status/rev-parse/ls-files per repo (both the ~ and
/root spelling), config is denied for those forms too, rm is denied and
notes are deleted through the vault's wiki-delete.sh, which validates the
slug. Verified headless: log allowed, config denied, push not allowed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The pass that verifies notes against sources, repos and the live cluster
and edits them still ran on the workstation timer, with no TypeSafe key,
no cost record, and no run on days the machine was off. It joins the
nightly pipeline after the audit: full clones of homelab and homeassistant
under ~/source, read-only kubectl via the ci-diagnostics token, its own
permission scope, JSON output so the report and total_cost_usd land in
the step log. push-notes also commits the rotation state and pushes the
agent's own snapshot commits.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every night with an empty queue started a full claude run that read the
directory and stopped, paying the agent's startup context and a model
pick for nothing (runs 10362, 10583).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The audit agent had no executable path in CI (Bash limited to the two
retrieval tools), wrote itself scripts it could not run and reported one
of seven checks (runs 10262, 10362). The checks are mechanical, so
notes/bin/wiki_audit.py does them without a model. push-notes adds only
wiki/, raw/ and output/, so scratch files in the checkout root are no
longer committed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The project moved to Codeberg; GitHub is a read-only mirror and the old
chart index at k8s-gateway.github.io returns 404, so the HelmRelease has
had no artifact. The chart is now published as an OCI artifact in the
project's own Codeberg registry (oci://codeberg.org/k8s-gateway/charts/
k8s-gateway), alongside the image.
OCIRepository + chartRef as for spegel and mariadb-operator; semver
range unchanged. Chart 3.7.2 -> 3.7.3, image moves to
codeberg.org/k8s-gateway/k8s_gateway:1.8.3. The rendered Corefile is
identical to the live one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
NETWORK-traffic-classes.md described table 48 and rules 980/981/990/991,
which #857 replaced with one table (47) decided by longest prefix, and
listed management as not started. Rewritten to the live state: five
classes, table 47 contents owned by bird, node IP and etcd on node0, the
etcd leave-and-rejoin procedure, and the mixed-state hazard hit during
the node IP move. The bird DaemonSet comment still named table 48.
Closes#576
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Records the live state after moving all three nodes:
- site-homelab.yaml: kubelet nodeIP.validSubnets fd02:46::/64 and
10.2.46.0/24 (was 10.2.40.0/24 and the legacy fd40::/64).
- cluster.yaml: etcd advertisedSubnets 10.2.46.0/24.
- mgmt0 comments: VLAN 40 is world-to-node only now.
Applied per node (k8s-3, k8s-2, k8s-1): drain, stage the config (delete
the old lists, set the new ones, one staged patch call), etcd
forfeit-leadership and leave, reboot. Each node rejoined etcd with its
VLAN 46 peer URL; EPHEMERAL was not wiped.
Refs #576
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Pod-sourced traffic is routed from table 47. Once the node IP moves to
VLAN 46 (step 3), pods reach node addresses there: the kubernetes
ClusterIP (socket LB rewrites it to the apiserver's advertise address),
hostNetwork pod IPs, and replies to kubelet probes. Without the attached
node0 route in table 47 those flows take the default to the UDM and every
in-cluster API client breaks. Taking it from the interface, like the
other attached networks, keeps the table in one owner.
Refs #576
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- node0: VLANConfig on the uplink trunk, VLAN 46, static fd02:46::1x/64
and 10.2.46.1x/24, accept_ra 0. Closed like pod0. Unused until the node
IP moves there (step 3).
- cluster.etcd.advertisedSubnets pinned to 10.2.40.0/24 so a reboot
cannot pick a node0 address: Talos never updates an existing member's
peer URLs.
Applied live with talosctl patch --mode=no-reboot, k8s-3 first. Node IPs
unchanged, etcd 3/3 healthy, node0 reachable node to node over v6 and v4.
Refs #576
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing selects cni.renner.uno/cilium-uninstall or
cni.renner.uno/multus-sole-writer since stage B (#915, #916). Removed
live from all three nodes with a $patch: delete strategic merge patch
(--mode=no-reboot); Talos dropped them from the Kubernetes nodes and
left the other labels in place.
Closes#832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cni.confPath: /etc/cni/net.d/multus mounts that host directory as the
agent's /host/etc/cni/net.d, so write-cni-conf-when-ready (unchanged,
.../05-cilium.conflist) lands in /etc/cni/net.d/multus/05-cilium.conflist,
the file the per-node CiliumNodeConfigs already produced on all three
nodes. The agent finds it unchanged and does not rewrite it.
updateStrategy maxUnavailable: 1 (chart default 2) so the roll takes one
node's datapath at a time.
bootstrap.sh renders the chart from these values, so a cold rebuild
lands directly in this layout.
Refs #832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
All three nodes run the sole-writer layout. Converge the Multus side and
drop the per-node Cilium overrides:
- kube-multus-ds-subdir becomes the only Multus DaemonSet, no node
affinity; kube-multus-ds (0 pods) and its ConfigMap are pruned.
- /etc/cni/net.d/multus is mounted DirectoryOrCreate so Multus waits for
the Cilium conflist instead of exiting on a missing directory.
- CiliumNodeConfigs removed. They only act when a cilium pod is created;
stage B2 moves the same path into the HelmRelease (cni.confPath) and
rolls the agents.
Refs #832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
k8s-1 carries cni.renner.uno/cilium-uninstall and
cni.renner.uno/multus-sole-writer, applied live with talosctl patch.
Verified on the node: /etc/cni/net.d holds only 00-multus.conf at the
top level, and after a reboot with immediate uncordon the OSD came up
with net1 (10.99.0.1) and no restarts. All three nodes are converted.
Refs #832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Deployment tolerated every NoSchedule taint (operator: Exists, no
key), including node.kubernetes.io/unschedulable. During the #832 node
reboots its ReplicaSet kept placing replacement pods on the cordoned,
shutting-down node; the kubelet rejected each with NodeShutdown, about
200 failed pods per reboot. Tolerate only the control-plane taint the
comment names.
Refs #832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
k8s-2 carries cni.renner.uno/cilium-uninstall and
cni.renner.uno/multus-sole-writer, applied live with talosctl patch.
Verified on the node: /etc/cni/net.d holds only 00-multus.conf at the
top level, and after a reboot with immediate uncordon the OSD came up
with net1 (10.99.0.2) and no restarts.
Refs #832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
k8s-3 carries cni.renner.uno/cilium-uninstall and
cni.renner.uno/multus-sole-writer, applied live with talosctl patch.
Verified on the node: /etc/cni/net.d holds only 00-multus.conf at the
top level, Cilium writes multus/05-cilium.conflist, and after a reboot
with immediate uncordon the OSD came up with net1 (10.99.0.3) and no
restarts.
Refs #832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing changes until a node is labelled:
- post-crds/cilium-nodeconfig.yaml: CiliumNodeConfig cni-uninstall
(label cni.renner.uno/cilium-uninstall) arms the agent's preStop
conflist cleanup; multus-sole-writer (label
cni.renner.uno/multus-sole-writer) moves write-cni-conf-when-ready to
/host/etc/cni/net.d/multus/05-cilium.conflist, a subdirectory libcni
does not read.
- multus/multus-subdir.yaml: kube-multus-ds-subdir with
multusAutoconfigDir /host/etc/cni/net.d/multus, scheduled only on
labelled nodes; kube-multus-ds gets the inverse affinity.
The kube-multus-ds template change rolls Multus node by node once on
merge.
Refs #832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kubernetes/infrastructure/netbird-proxy-services -> netbird-service-sync:
SA, ClusterRole(Binding) and Deployment renamed, in namespace
netbird-service-sync, image gitea_admin/netbird-service-sync pinned to
the master build digest, BAO_ROLE=netbird-service-sync.
- Old policy doc netbird-proxy-services.hcl removed; Traefik comment
follows the name.
The infrastructure KS owns both old and new objects, so prune and apply
happen in one pass. OpenBao policy and role netbird-service-sync are
already written.
Closes#876
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Additive; the running controller is untouched:
- ci/images/netbird-proxy-services -> ci/images/netbird-service-sync,
build step and image name renamed, so the merge builds
gitea_admin/netbird-service-sync.
- Namespace netbird-service-sync with a baseline policy: probes in;
out only to kube-dns, kube-apiserver, OpenBao :8200 and world :443
(NetBird management API).
- openbao-ns admits netbird-service-sync on :8200, like wiki-index.
- policies/netbird-service-sync.hcl documents the new policy and role
(applied by hand, as before).
Refs #876
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Gitea runs in the gitea namespace since #906. Remove what only served it
in apps:
- openbao rotator egress (flux-git, review-bot): apps/gitea target.
- ai-ns wiki-index ingress: apps/gitea source.
- apps default-deny egress to wiki-index :8080 (gitea's webhook; now in
gitea-ns.yaml).
- Velero schedule comments.
Refs #878
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kubernetes/apps/gitea: every object in namespace gitea; applied by a
new gitea Flux Kustomization (clusters/homelab/gitea.yaml) instead of
apps. gitea-allow-metrics-scrape dropped: the gitea-ns baseline
already admits monitoring.
- Rotators (flux-git, review-bot) and the Prometheus gitea job use
gitea.gitea.svc.
- bootstrap.sh waits for the gitea KS.
Data: data-gitea-0 is not copied. The PV (reclaimPolicy Retain) is
rebound to a PVC gitea/data-gitea-0 pre-created with volumeName, which
the new StatefulSet adopts. gitea-tls is pre-issued in the gitea
namespace so HTTPS has no certificate gap.
Refs #878
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Additive only, nothing moves yet:
- Namespace gitea, declared in infrastructure like woodpecker.
- network-policies/gitea-ns.yaml: baseline (same-ns, monitoring, host,
ICMP; egress to traefik, kube-dns, wiki-index, world) and
gitea-allow-traefik for :3000 and :2222.
- openbao rotator egress (flux-git, review-bot) and wiki-index ingress
also admit gitea in the gitea namespace; the apps entries stay until
stage 3.
- Velero schedule gitea-daily (07:45) so data-gitea-0 stays backed up
after the move; apps-daily comment updated (Woodpecker left apps).
Refs #878
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
decide required every consumer to pin the same ci-claude tag and aborted
otherwise (pipeline 10497: [2.1.250 2.1.278]), which needed a hand fix
before the latest version could roll out. Record all distinct tags, bump
whenever any is off latest, and rewrite every version-pinned tag to the
new one with a validated x.y.z version.
Merging this is itself a push touching the pipeline file, so the refresh
runs, builds the latest image, and opens the bump PR.
Refs #889
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Pipeline 10186 failed in bump-pr with 'url contains a newline in its
username component': ci_claude_bump_bot_token carries a trailing newline,
so the push URL (and the PR API auth header) broke. Strip CR/LF once at
the top of the step and fail fast if the token ends up empty. This makes
the step independent of how the secret value was written.
Closes#889
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Unset, paperless derives cpu_count / TASK_WORKERS, i.e. 20 parallel
ocrmypdf workers on k8s-1. Memory during OCR scales with that count, so
the limit alone is not enough headroom for large scans.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An 18 MB scan from the ADS-1700W has kept the paperless container in an
OOM crash loop since 2026-09-21 10:21: ocrmypdf runs with 20 workers
(paperless default, cpu_count / TASK_WORKERS on a 20-core node) and blows
the 2Gi limit ~40s into OCR. The consume task is re-queued on every
restart, so the same file kills every attempt. 117 restarts so far, web
UI down as a side effect since worker and webserver share the container.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 10254: POST .../merge with delete_branch_after_merge returned
HTTP 500 although PR #2 on k.renner/notes was merged, so the step
failed and left the branch behind. The merge call no longer asks Gitea
to delete the branch; the step reads the PR back and requires
merged=true, then deletes the branch as a separate best-effort call.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First live pick after #893 chose claude-sonnet-4-5 at 0.23 over
claude-sonnet-4-6: same-family models split the probability for a
mid-weight task, and "prefer newer" is a policy, not a judgment. New
scripts/ci/choose-model.sh applies it to the distribution: among
candidates within CHOOSE_MODEL_MARGIN (0.7) of the best probability the
most recently released wins; a clear best stays; NONE yields nothing.
The picker logs the top probabilities. Four policy tests added.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The picker ran a full claude -p in plan mode per selection and parsed
JSON out of prose. With TYPESAFE_API_KEY set it now sends the task and
the catalog rows as state and asks one Choice over the model ids, each
with its catalog row as criterion plus NONE; the id is validated against
the pristine catalog as before. No key or a failed request falls through
to the claude picker unchanged.
Measured on the two-model fixture (Opus 5 / Haiku 4.5): issue-fix and
security-fix -> Opus 0.99, alert triage -> Opus 0.62, wiki audit ->
Haiku 0.94, update audit -> Haiku 0.46, distill -> Haiku 0.63; about a
second per pick. Dry-run test added to test-model-selection.sh.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Instead of rebasing onto a moving main: the run's changes go to
wiki-maintenance/<date>-<run>, a PR is opened and merged via the API
(Do: merge, branch deleted afterwards). A merge conflict leaves the PR
open for a human and fails the run. Fields checked against the
instance's swagger.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 10241: distill and audit succeeded (5 sources, 8 new + 6 updated
notes), the push was rejected non-fast-forward because the workstation
pushed the vault while the agents ran. The push step now fetches,
rebases and retries; a real conflict still fails the run.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A deployment-event run (10239) errored before its first step: the
typesafe_api_key secret is scoped to cron and manual only. Manual
pipelines (woodpecker pipeline create) are the way to start this by
hand; the deployment trigger is dropped.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First cron run (10220) failed in checkout-notes with "could not read
Username": the wiki_bot_token secret had been set from a file with a
trailing newline, which git carried into the extraheader. The secret
is rewritten without it; the pipeline now strips CR/LF from the token
and the index credential regardless of how they were entered.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The refresh triggered by #888 built 2.1.278 (the first ci-claude with
python3) rather than rebuilding 2.1.250, and its bump-pr step failed on
a newline in the push token secret (#889), so the consumer bump never
came. This pipeline needs python for wiki_ask.py, so it moves ahead of
the general bump.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Replaces the workstation's wiki-maintenance.timer. The pipeline clones
k.renner/notes, runs wiki-distill over raw/ and wiki-audit over wiki/
as two separate `claude --agent` runs (no background tasks in print
mode, which killed the workstation job at the 600 s ceiling), then
commits and pushes as Claude Bot; the push webhook refreshes wiki-index.
The agent definitions now live in the vault (.claude/agents/), so CI
and workstation share them. The agents and their retrieval scripts
address the vault as ~/Notes; each agent step links $HOME/Notes to the
checkout. ci-claude gains python3 and py3-yaml for wiki_ask.py.
Secrets on the repo: claude_code_oauth_token (existing), typesafe_api_key,
wiki_bot_token (claude-bot, write:repository), wiki_index_basic_auth.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
wiki-index:2026-09-20-4b529cd1 is the build of the previous commit,
i.e. master's code with the paging fix and /ready. Manifest-only
commit, so it does not trigger another build.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three builds on 2026-09-20 (two branch pushes, the merge) rewrote
wiki-index:2026-09-20 in an order the digest pin could not follow: the
pinned digest ran the paging-only build without /ready and crash-looped
on its startup probe. A <date>-<sha8> tag is immutable per commit and
lets the deployment pin by tag without a registry lookup.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Same dated tag again; the pod holds the #879 build. Pinning the new
digest (paging fix, /ready readiness) rolls it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
/health answered 503 until an index existed and fed startup and
liveness, so a live process without an index counted as dead and was
restarted every 15 min, and a later failed sync would have pulled a pod
with a good index out of the Service. /health is now always 200
(process serves); /ready is 200 once an index exists and stays so
across failed syncs, which keep the last good index. Probes follow:
startup 60 s on /health, readiness on /ready, liveness on /health.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The notes repo tree has 3192 entries. Gitea reports truncated=true on
every page including the last one (page 4, 192 entries) and answers
"tree": null one page further, which the sync hit:
"sync failed: 'NoneType' object is not iterable". Paging now stops on a
short page or when total_count is reached, and a null tree is empty.
Unit test with the observed page shapes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#879 rebuilt wiki-index under the same dated tag 2026-09-20; the node
still holds the first build (sha256:085438d8) and IfNotPresent would
not re-pull. Pinning tag@digest rolls the exact new image.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`bao kv put key=@file` stores the file's trailing newline; the first
sync after #875 failed with "Invalid header value b'token ...\n'" and a
webhook signed with the newline-free secret would never have verified.
Credentials are stripped on read. A failed sync now retries after 60 s
instead of waiting the full RESYNC_SECONDS.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The apps namespace is egress default-deny; gitea's webhook to
wiki-index.ai.svc.cluster.local:8080 was dropped (a test delivery
produced no request at the service). Namespace-wide egress rule scoped
to the wiki-index pod on 8080, the mosquitto pattern; wiki-index's own
ingress policy already admits only the gitea pod from apps.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The openbao namespace policy admits only its own pods, Traefik and
External Secrets. wiki-index (ns ai) logs in through the Kubernetes auth
role on every sync and timed out on the first attempt:
"POST .../v1/auth/kubernetes/login: timed out". Pod-scoped ingress rule
for it, mirroring the external-secrets entry.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deployment + Service + PVC + ServiceAccount in namespace ai, LAN-only
Ingress wiki-index.renner.uno behind Traefik BasicAuth (ExternalSecret
from OpenBao, same shape as wiki-embed), Flux Kustomization depending on
llama-server (wiki-embed is the embedding backend), pod-scoped
CiliumNetworkPolicy admitting Traefik and Gitea (push webhook) on 8080
and allowing egress to OpenBao 8200, and the OpenBao policy for the
service's Kubernetes auth role.
Image: gitea.renner.uno/gitea_admin/wiki-index:2026-09-20, built by the
step added in the previous PR; the tag is the build date.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Runtime image for a wiki-index service in namespace ai (manifests follow
once the image exists): standard-library Python that mirrors the notes
repository's wiki/ from Gitea into a bge-m3 index and serves POST /rank
to ~/Notes/bin/wiki_rank.py.
Sync fetches the branch head and tree via the Gitea API, embeds only notes
whose blob sha changed (through the in-cluster wiki-embed), drops notes
that left the tree, and persists the index on a PVC. It runs at start, on
every push webhook (HMAC-SHA256 verified) and every RESYNC_SECONDS. Gitea
token and webhook secret come from OpenBao through the Kubernetes auth
role on every use. Five unit tests cover parsing, incremental sync,
persistence, ranking and error handling.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The wiki-embed-basic-auth ExternalSecret from #871 never synced:
"using cluster store openbao is not allowed from namespace ai: denied by
spec.condition". The store's namespace allowlist predates the ai
namespace. Adding it, sorted, as the manifest's own maintenance note
prescribes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Second bge-m3 llama.cpp Deployment in ns ai so wiki queries never queue
behind Paperless on llama-embed. Shares the llama-server-models PVC, so
the weights are not downloaded again.
Exposed LAN-only as wiki-embed.renner.uno (no dns.renner.uno/public
label) for bin/wiki_rank.py on the workstation and the nightly wiki jobs.
The client is a script, and the Authentik gate is browser-only, so the
Ingress uses Traefik BasicAuth backed by an ExternalSecret, the same
shape as loki.renner.uno. A pod-scoped CiliumNetworkPolicy admits Traefik
to wiki-embed only; llama-server stays reachable solely via its gate.
Measured on 40 past wiki-synth queries: embeddings picking 150 candidates
plus one TypeSafe Choice puts a cited note in the top 5 for 95% of
queries at a sixth of the cost of ranking all 1700 notes with TypeSafe.
Needs before first use: OpenBao key apps/ai/wiki-embed-basic-auth with
field `users` (htpasswd line), and the same credential as user:password
in ~/.config/wiki-embed/basic-auth on the workstation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Einzelraumsteuerung (8 radiator valves, 9 climate sensors, 9 underfloor
actuators, 3 floor thermostats) runs on Viessmann's One Base ZigBee system.
Its room and device registries and its live per-device values sit on ECU 0x6c3
(BACKENDGATEWAY) and are readable in plaintext over the E3 bus -- room
temperature, the active target temperature, humidity, valve opening, demand,
battery and RSSI. That replaces the ViCare cloud room sensors with a local
source.
0x6c3 had never been enumerated: only 0x680, 0x6a1 and 0x68c ever had a full
scan. A full read of 0x6c3 returns 155 decodable datapoints plus 35 that exist
but fail the stock codec.
Those 35 are the reason for the overlay. open3e declares
ZigBee<N>DeviceCurrentValues as 57 bytes; a PAIRED slot answers with 68 -- the
known 57 plus the parent device serial and three trailing bytes. udsoncan reads
only the declared length and then treats the next two bytes, the first half of
that serial, as a foreign data identifier, so every paired slot fails with
"Server returned values for data identifier 0x.... that was not requested". The
bogus identifier is stable per device and differs between devices because the
serial does, which is why this previously read as an encrypted channel. It is
not encrypted.
The overlay re-declares the 29 paired slots at 68 bytes and inherits upstream's
sub-type list, so the per-device-type switch -- including
DeviceTemperatureSetpoint -- comes from upstream rather than being copied.
Empty slots answer with 57, decode against the stock codec and are not polled.
Only CurrentValues are polled. The registries are static configuration and are
read on demand.
Verified against payloads captured from 0x6c3 on 2026-09-17, decoded with
open3e 0.7.5 (the commit pinned in the heating-can image): all 29 slots are 68
bytes, and a radiator valve yields ActualTemperature 19.3 °C, ValveOpening 0 %,
DeviceTemperatureSetpoint 8.0 °C, battery 38 %, RSSI -69 dBm. The setpoint
field was confirmed by changing one room in the ViCare app three times and
re-reading: 8.0 -> 15.0 -> 13.0.
Read-only. No write path to the per-room setpoint is known; writes to the
CurrentValues DID were attempted and had no effect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ServiceMonitor added in #865 discovers both external-dns targets and
then times out on them: the apps namespace is default-deny-ingress, so the
scrape never arrives.
lastError: Get "http://10.244.2.218:7979/metrics": context deadline exceeded
health: down
Same shape as gitea-allow-metrics-scrape, and one policy covers both
releases — the chart gives them the same app.kubernetes.io/name and they
share the namespace, even though external-dns-unifi is declared under
infrastructure/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
external-dns metrics say what the publisher intends. This says what a client
actually gets, and catches causes the publisher cannot see — a record removed
on the UDM directly, for one.
Two findings shaped it.
The probe asks the UDM directly rather than the cluster resolver. k8s-gateway
caches these names with `serve_stale 1h immediate`, so a deleted record would
keep answering from cache for an hour before the probe noticed.
And rcode alone is not a usable signal here. Measured against 10.2.40.1:
does-not-exist-xyz.renner.uno -> NOERROR, CNAME proxy.renner.uno
does-not-exist-xyz.paperless.renner.uno -> no answer
The public *.renner.uno wildcard matches one label deep, which is exactly
where ha, mqtt, ldap and syslog live: delete their records and the UDM still
answers NOERROR, pointing at the WAN proxy. A probe checking only the rcode
would have stayed green through the outage for four of the six names. So the
modules require an A record in the answer and reject a CNAME, which separates
a real record from the wildcard without encoding any address.
One module per name because the dns prober reads query_name from the module
and takes the resolver as its target; both families are probed per name and
the alert wants both to fail, so the UDM's bursty connection refusals do not
read as a missing record. That is also why the window is 30m rather than the
10m the HTTP probes use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither external-dns release was scraped: both expose /metrics on :7979 and
no ServiceMonitor existed, so external_dns_* series were absent from
Prometheus entirely. The process that publishes every LAN name in the
cluster had no instrumentation at all.
That is how 2026-09-12 went unnoticed for five days. external-dns v0.22.0
changed the default annotation prefix, the service source went empty, and
policy=sync deleted six records. source_endpoints_total fell 48 -> 36 and
controller_verified_records 24 -> 18 per family; nothing was watching either.
ExternalDNSSourceEndpointsDropped is the rule for that shape: a sustained
drop below 90% of the 24h peak. The threshold is chosen against the observed
inventory — one retired Ingress is 2 endpoints of ~48 and stays quiet, a
whole class of names does not. A deliberate mass removal fires it once and
clears itself after 24h, which is the right trade for a signal that
otherwise only speaks when records are already gone.
The rest of the group covers the ways the drop rule itself can go blind:
source and registry errors, the instance being down, and an absent() dead
man mirroring BlackboxProbesAbsent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
external-dns v0.22.0 changed the default --annotation-prefix from
external-dns.alpha.kubernetes.io/ to external-dns.kubernetes.io/ with no
fallback, and its release notes lead with the consequence: "This change can
delete all your DNS records."
The chart floated onto it at 2026-09-11 23:47:56 (appVersion 0.21.0 ->
0.22.0). Two hours later, at 2026-09-12 01:47:59, the unifi provider logged
`client deleted record` for every name that depended on the annotation:
ha.renner.uno mqtt.renner.uno ldap.renner.uno
syslog.renner.uno ftp.paperless.renner.uno
syncthing.paperless.renner.uno
A, AAAA and the TXT ownership records, and no `client created record` since.
Every one of those is a LoadBalancer Service, whose hostname external-dns
can only learn from the annotation — so the source went silent and
--policy=sync removed the records. Ingress-sourced names take their host
from spec.rules[].host and were untouched, which is exactly the split
observed: source_endpoints_total is 36 with source_errors_total 0.
These six annotations were the only alpha-prefixed external-dns annotations
in the repo, so moving them to the current prefix is the whole fix. Pinning
the prefix back with --annotation-prefix would keep us on a deprecated
default that upstream intends to drop.
Scanning has been broken since: the Brother resolves ftp.paperless.renner.uno
rather than using the address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LB address was declared twice: on the Service as io.cilium/lb-ipam-ips,
and again as a literal PASV_ADDRESS in the ConfigMap generator.
The copy was not doing anything. vsftpd listens on a single IPv6 socket, so
an IPv4 client arrives as ::ffff:a.b.c.d and vsftpd answers PASV with
0.0.0.0 regardless of the configured address — the only such session in 30
days of logs got `227 Entering Passive Mode (0,0,0,0,82,109)` while
PASV_ADDRESS=10.2.99.82 was set. IPv6 clients negotiate EPSV, which carries
a port and no address at all; that is what the scanner uses.
Removing the literal falls back to the image default of 0.0.0.0, which is
the reply clients already receive, so behaviour is unchanged and the address
now exists in exactly one place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vsftpd 3.0.5 segfaults its standalone listener on roughly 0.3% of the
connections it accepts: the parent closes the accepted socket before it
registers the forked child's pid, so a child that exits inside that window
makes the SIGCHLD path look up an unregistered pid, get NULL, and
dereference it in hash_ip(). PID 1 is vsftpd, so the segfault takes the
container with it.
The tcpSocket probes added in 469cdbc2 opened ~8600 connections a day
against that, which turned a latent crash into a restart every ~25 minutes
and left the scanner hitting CrashLoopBackOff windows. Measured on the live
pod: 52 restarts in 21h, one node-side `vsftpd[...]: segfault at 0` per
restart, starting the hour the probes rolled out.
Assert the listening socket from inside the container instead. The probe
still catches the failure it was added for — vsftpd not serving port 21 —
without being a client of the process it watches.
vsftpd 3.0.5 is upstream's last release (2021) and Alpine ships it
unpatched in 3.21 (3.0.5-r2) and edge (3.0.5-r3), so there is no version
bump that fixes this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TokenStorageMethod is Encrypted and the container has no OS secret store,
so LibationCli falls back to minting a portable master key at
/config-internal/libation-master.key -- a path that is not a mounted
volume. The key is therefore new on every run and discarded at exit,
which is why the entrypoint logs LAST-RESORT on each pass.
The entrypoint looks for the key at ${LIBATION_CONFIG_DIR}/
libation-master.key, i.e. on the PVC, and copies it into place when
found. The key is now stored in OpenBao alongside the other bootstrap
files, so seed it like them: without it a rebuilt PVC would hold tokens
encrypted under a key that no longer exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image entrypoint stages config into /config-internal, which is not a
mounted volume, and it does so by copy:
init_config_file() {
...
cp ${FULLPATH} ${LIBATION_CONFIG_INTERNAL}/
Only LibationContext.db is symlinked to the PVC. AccountsSettings.json is
therefore read from the PVC at startup and written back nowhere, so every
token refresh Libation performs is discarded when the container exits. On
the live PVC the file still carried its 2026-05-14 seed timestamp, four
months after deployment, while the DB beside it was current.
Pass args instead of letting the entrypoint run its own loop. The
entrypoint completes its setup and then execs what it is given, so the
staged environment is intact; an EXIT trap copies the refreshed file back
to the PVC whether the run succeeds or fails.
This was found while recovering from Audible refusing content licenses
for this account. Re-registering the account fixed the refusals; without
this change the fresh registration would have been discarded on the next
run exactly like every refresh before it.
Also corrects the init container comment, which claimed the sentinel
preserved token refreshes. It preserves the DB; it never preserved the
tokens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CronJob ran `rmcrackan/libation:latest` with imagePullPolicy Always
but was stuck on 13.7.11 (2026-08-24). Spegel answers containerd's tag
resolution from its peer cache -- a 200 from the mirror is a hit, so
upstream is never consulted and the tag never advances. Verified: the
pod's imageID was sha256:946177d3 (= tag 13.7.11) while Docker Hub's
:latest pointed at sha256:6f9d8ba1 (= 14.2.1), and querying a Spegel
pod's registry port directly returned the stale digest for that tag.
Audible changed device registration in early September; upstream shipped
an experimental fix in 14.1 and the real fix in 14.2, so the stale image
predates both. Since 2026-09-15 three newly purchased titles fail with
"Content license denied" and the library has not grown.
A digest reference is immune to the mirror's tag cache. Note that the
upstream maintainer's remedy also requires removing and re-adding the
Audible account with an interactive login, which this change does not
and cannot perform.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measuring VLAN 40 for the traffic-class split turned up a share of the
management segment that is not management: the node's Ceph client
talking to OSDs on the other two nodes, sourced from the node address
on mgmt0.
Ceph sends every client read to the acting primary of the PG, which
with three OSDs is remote two times out of three. Read affinity has the
client read from the replica in its own CRUSH location instead.
The preconditions hold here. replicapool and cephfs/data0 are size 3
over failureDomain host across three nodes, so every object has a copy
on every node. The OSDs are created under zone buckets
(root=default host=k8s-N zone=<name>) and the nodes carry
topology.kubernetes.io/zone, which is in Rook's default
crushLocationLabels. Ceph is v20.2.4 and the kernel 6.18, against a
documented minimum of 5.8.
Writes are unaffected: they always go to the primary, and the
replication behind them already rides the VLAN 49 cluster network.
cephfs-ci is size 2, so a local replica is not guaranteed there and the
client falls back to the primary.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rolling out #857 took cross-node pod traffic down, and cluster DNS with it, until it was reverted. The table was incomplete: the node's own pod CIDR was originated as a blackhole so it could be advertised without being installed, and then filtered out of the kernel export.
The rule matches the SOURCE, and a reply from a pod on another node is forwarded traffic with a pod source — so it matches too. Its destination is a pod on this node, the table had no route for that, and it took the default and left for the UDM. The comment claiming such traffic would fall through to main was wrong: nothing falls through once a default matches.
Originating it via cilium_host instead makes it both advertisable and correct in the table, and the export filter that existed only to hide the blackhole goes with it.
Diagnosed by narrowing the rule to a single pod address, where the same path worked because the return packet no longer matched: 404 with src <pod>/32, 000 with src 10.244.0.0/16, 404 again once 10.244.0.0/24 dev cilium_host was added to table 47 by hand. bird -p parses the corrected config inside the image, exit 0.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule stack grew a carve-out layer because table 47 held nothing but a default route: it matched everything, so every cluster-internal destination had to be excluded by a rule naming its prefix. Those exclusions were wrong twice over — too broad, since fd00::/8 and 10.0.0.0/8 also caught every LAN client and sent its reply back over mgmt0, and unnecessary, since the networks in question are directly attached and a connected route already beats a default.
So the table gets the connected routes and decides on its own. bird's direct protocol generates them from the interfaces rather than from a list, alongside the peers' pod CIDRs it already learns and the egress default it now originates. One rule per family remains, on the pod source prefix; 980/981, 990/991 and table 48 are gone.
bird owns the table outright, including the default. That is what keeps the failure benign: if the daemon stops the table empties, the rule finds nothing and falls through to main, so pod traffic returns to VLAN 40 unseparated rather than misrouted. A default declared in the Talos config would have outlived the specific routes and sent pod traffic to the UDM.
The ingress gateway moves from a route in each node patch to one node annotation in the site patch, which is now the only place it is stated.
Verified: bird -p parses the rendered config inside the image, exit 0; the templates render GW6 fd02:47::1 and GW4 10.2.47.1 from the live k8s-1 object; kubectl kustomize builds.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the proposal, which was wrong before it was written down. It presented A, B and C as the option space and concluded pod-to-pod was blocked; policy routing was available the whole time and is what moved pod egress and then pod-to-pod. The file recorded a blind spot and gave it the authority of a checked-in document.
Three of four classes are separated now, so this describes the built design rather than proposing one, and the PROPOSAL- name goes with it: which link each class uses, the rule order and why 980/981 match source and destination, where each table comes from, how the bird mesh renders itself from the API without repeating an address, and why the node's own CIDR is originated rather than read back from the kernel.
Management is the remaining class. Its levers are listed, along with the two approaches the owner rejected — with the caveat that the same reasoning which produced that list also called pod-to-pod impossible, so it is a record of decisions and not an analysis to trust.
Failure behaviour is stated because it is what makes the design safe to roll one node at a time: an empty table 48 falls through to main and pod traffic returns to VLAN 40, unseparated rather than broken.
bootstrap.sh's reference follows the rename; its point was always that the QEMU/Hetzner sites use two segments, not the homelab's split.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last step of the traffic-class split. The bird mesh already fills table 48 with one route per peer pod CIDR, next hop the peer's pod0 address; these two rules are what make the kernel look there.
Source AND destination are matched. Source alone would also catch the node's own traffic to a pod, which would then leave with a pod0 source — an address CiliumNode does not list, so Cilium would classify it as world rather than host and policy would drop it. Destination alone would catch traffic from anywhere.
Priority 980/981 sits ahead of the 990/991 pair that sends the cluster and local networks to main, so pod-to-pod wins for that traffic and everything else is unchanged. Traffic to a pod on the same node falls through: table 48 holds only the peers' CIDRs, the lookup misses, and 990/991 send it to main and cilium_host.
Verified on k8s-1 with --mode=try, 200 requests per family from a pod on k8s-1 to a pod on k8s-2: pod0 gained 6957 transmit packets, matching 400 requests, while pod0 receive moved by 45 — the peer is unpatched and still answers over mgmt0, the same asymmetry the earlier measurement showed. Both rules were installed as expected, one per family.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IPv6 populated table 48 on all three nodes; IPv4 was empty. The daemon log gives the reason: KRT: Received route 10.244.0.0/24 with strange next-hop 10.244.0.195. Cilium's IPv4 pod route points at a next hop inside the prefix it describes, which BIRD's kernel protocol refuses. The IPv6 route is a device route with no next hop, so it passed — the asymmetry, not a filter, is why one family worked.
Reading the route back made BIRD's view depend on the shape Cilium happens to install. The prefix is already in the Node object, so a static protocol originates it directly: blackhole, because the route exists only to be advertised, and the sole kernel protocol exporting from these tables takes BGP routes.
That removes the kernel-learning tables, the pipes and the own-CIDR filters entirely — with them the dependence on learn all, on ifname, and on the maximum-length exclusion.
Verified: the templates render OWN_POD4 10.244.0.0/24 and OWN_POD6 fd10:244::/64 from the live k8s-1 object, and bird -p parses the result inside the image with exit 0.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DaemonSet came up with Init:Error, kubectl: not found. The build had pushed sha256:3bc85006 for bird:2026-09-16, but every pod resolved that tag to sha256:75e19214 — the first build of the same day, from before kubectl was added.
imagePullPolicy: Always does not help: it re-pulls, but the tag itself resolves to the stale digest. Deleting the pods changed nothing; three fresh pods on three nodes all resolved to the old digest. A one-off pod pulling the digest directly ran kubectl 1.33.1 and BIRD 2.17.2, so the registry holds the right image and only tag resolution is stale.
The dated tag cannot carry a same-day rebuild, and this image was rebuilt three times today. Referencing the digest makes the deployment deterministic. The cost is that the reference has to be updated whenever the image is rebuilt; the alternative, tagging by commit, does not work for a manifest living in the same commit.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing reads table 48 yet. The mesh fills it so the routing rules land on a table that is already correct, and so a mistake here shows as an empty table rather than as broken pod traffic.
No address is written down twice. Each node publishes its pod0 addresses as Talos nodeAnnotations, beside the VLANConfig that assigns them; the init container reads the node list and renders from it. bird.conf holds no address at all, and the pod CIDRs are not restated either — the own-CIDR filter keys on the outgoing interface plus a maximum-length exclusion, which separates the node's pod route from cilium_host's own host address without naming a prefix.
Rendering is go-template-file, not string building: kubectl renders the two templates, a field selector drops the node itself server-side, and in-cluster config removes the token and CA plumbing. kubectl comes from the Alpine package, so there is no version literal to drift out of sync and no download to trust; it only ever reads core/v1 Nodes. jq and curl are gone.
The init container ends with bird -p, so BIRD's own parser gates startup: a node whose annotation is missing renders as <no value> and fails there rather than crashlooping the daemon or forming a mesh with a hole.
learn all rather than learn, since plain learn skips proto kernel. direct is explicit because iBGP defaults to multihop, which would make the gateway recursive. strict bind keeps the listener off mgmt0 and ingress0.
Verified against the live cluster with all three nodes annotated via --mode=try: the templates rendered the expected defines and four peer protocols, and bird -p parsed the result inside the image with exit 0 — which also confirms net.len and the include ordering. kubectl kustomize builds.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-node peer list is rendered from the Kubernetes API rather than written into the repo a second time, so the init container needs an HTTP client that can verify against the API server's CA bundle. busybox wget cannot, and the query is not going to run unverified. Using the same image for the init container and the daemon keeps one lifecycle instead of borrowing a CI image at runtime.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every other Alpine image under ci/images/ pins alpine:3.21 by index digest, and alpine-jq states the reason in the file: the resolved version drifts between restarts of an otherwise pinned image. The first revision used a bare alpine:3.22 tag. 3.22 is still right — bird2 first appears in 3.22/community — so the tag stays and the index digest is added.
heating-can runs --help checks before publishing so a wrong path fails the build rather than the first pod; bird documents --version, so it gets the same guarantee.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod east-west follows the peer's Kubernetes InternalIP on mgmt0, and Cilium offers no way to point it elsewhere — established by test in #838, and directRoutingDevice (#837) does not do it. Policy routing does: a rule on the pod source prefixes into a table whose routes carry pod0 next-hops, proven on k8s-1 by counting 4647 packets onto pod0.
Those routes have to be generated, not written down: PodCIDRs are Kubernetes-assigned and do not survive a rebuild in place. BGP between the nodes over pod0 generates them, with the session source as the next-hop.
BIRD rather than FRR. FRR's docs describe 'set table' only as setting a BGP table identifier, never a kernel table; binding bgpd to an address implies --no_kernel and would suppress the route installation the design exists for; and a gatewayless IPv6 device route has no next-hop for its route-maps to match, so the own-CIDR filter could not be uniform. BIRD documents 'kernel table', 'strict bind' without touching the kernel protocol, and an 'ifname' filter attribute that discriminates the local route from the learned ones in both families.
No maintained standalone BIRD image exists: pierky/bird self-declares as lab-only and amd64-only, calico/bird is a Calico-internal artifact expecting confd, slipmesh/bird is five weeks old, srl-labs and ibhde are lab-scoped or on BIRD 1.6. A custom controller would have needed an image of its own anyway, so the build is common to both routes.
Image only. The DaemonSet is 2/3 and the routing rules 3/3, so the table fills before anything reads it.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default route cannot simply move to ingress0: the node IP, the API VIP and apid answer on mgmt0, so a reply leaving over ingress0 would carry an mgmt0 source onto VLAN 47. Policy routing splits the two instead — pod source prefixes look up table 47, everything the node sends itself keeps main and mgmt0.
Table 47 holds only the two default routes, so destination rules at 990/991 send the cluster and local networks back to main first; without them a pod's packet to a service matches the source rule, finds only the default, and leaves for the UDM. That was measured on k8s-1, where in-cluster DNS stopped resolving until the rules were added.
The v6 gateway is the UDM's configured ULA on the segment, the address BGP already peers with, rather than the RA-learned link-local. accept_ra_defrtr stays 0 on ingress0 so the RA default does not land in main.
Verified on k8s-1 with --mode=try: a pod egressed as 2003:d6:973f:5605:5a47:caff:fe74:7027, the ingress0 GUA from the VLAN 47 prefix, while in-cluster DNS and the Talos and Kubernetes APIs kept working.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pod-to-pod traffic currently rides the management VLAN. In tunnel mode
Cilium encapsulates to the peer's Kubernetes InternalIP, so east-west
follows VLAN 40 wherever it goes -- which blocks moving management onto
the 1G NIC without dragging pod traffic with it.
The node IP stays on VLAN 40. Instead this uses the documented device
selection: native routing hands forwarding to the kernel, and
autoDirectNodeRoutes installs the remote PodCIDR routes, which is
permitted because all three nodes share pod0's L2 segment.
"By default, if a single device was detected or specified via devices
then Cilium will use that device for direct routing. Otherwise,
Cilium will use a device with Kubernetes InternalIP or ExternalIP
set."
directRoutingDevice is therefore set explicitly: mgmt0, ingress0 and
pod0 all carry addresses, so the fallback would pick mgmt0.
Verified beforehand that pod0 is a working path between nodes -- from a
pod on k8s-2, k8s-1's kubelet answers 401 on both 10.2.48.11:10250 and
[fd02:48::11]:10250, so the L2 and both address families work.
ipv4/ipv6NativeRoutingCIDR are already set and become mandatory in this
mode; 10.244.0.0/16 and fd10:244::/48 sit inside them, so pod-to-pod
stays unmasqueraded as before.
Not rolling: dropping encapsulation is a cluster-wide datapath change
and a node that has switched cannot exchange pod traffic with one that
has not. Needs a scheduled window, not a background reconcile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Phase B step 2 of the traffic-class split. The eBGP session source
address is what the UDM installs as the next-hop for every PodCIDR and
LoadBalancer prefix the cluster advertises. Peering over the management
LAN therefore pinned all world->pod return traffic to the 1G mgmt link,
which is the thing #576 exists to fix. Moving the peers onto VLAN 47
puts that traffic on the 10G ingress link instead.
No localAddress or sourceInterface override is needed: the kernel
already picks the ingress0 address for both families.
ip -6 route get fd02:47::1
fd02:47::1 dev ingress0 proto kernel src fd02:47::5a47:caff:fe74:7027
Both peer addresses were verified reachable from the cluster before
this change -- 10.2.47.1 and fd02:47::1 each answer HTTP with a 301,
and the UDM's neighbour entry on ingress0 is REACHABLE.
The UDM already accepts sessions from this segment: PR #835 added the
10.2.47.0/24 and fd02:47::/64 listen ranges and is uploaded. Its VLAN 40
ranges stay in place for now and get dropped in step 3, once every
session is confirmed established on 47.
gracefulRestart (120s) covers the peer replacement, so the UDM holds the
learned routes across the flap rather than withdrawing the LB VIPs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First step of moving the world-to-pod class onto ingress0 (#576). This
changes nothing about where the cluster peers from — it only widens
what the UDM will accept.
The order matters. Cilium's sessions are outbound to the UDM, and the
UDM accepts them through `bgp listen range`. Moving the cluster to
ingress0 before the UDM listens on 10.2.47.0/24 would drop every
session, and with them every advertised LoadBalancer VIP, so external
ingress would go down until the UDM caught up. Listing both segments
first removes that window entirely: a listen range only accepts a
session, it never opens one, so the 47 ranges sit idle until the
cluster actually moves.
The prefix-lists and route-maps are untouched. They filter which
prefixes the UDM accepts — pod CIDRs and the LB pool — and the cluster
advertises the same prefixes either way. Only the next hop changes.
Uploaded by hand at UniFi UI -> Settings -> Routing -> BGP, as before.
Verify with `vtysh -c "show ip bgp summary"`: the six existing sessions
on 10.2.40.x and fd40:: must still be established afterwards, and no
new ones appear until the cluster side moves.
The 40 ranges come out in a later change, once every session is
established on 47.
Refs #576
Adding a second DHCP client per node in #831 broke node DNS. Both
clients announce the same hostname, so the UDM registered each node
against VLAN 47 instead of VLAN 40: `k8s-1.srv.renner.uno` and
`k8s-3.srv.renner.uno` stopped resolving, and the reverse for
10.2.47.11 came back as bare `k8s-1`. The lease's resolver also
displaced the management one, leaving 10.2.47.1 ahead of 10.2.40.1 in
the node's DNS servers.
That matters beyond `talosctl -n <fqdn>`: bootstrap.sh resolves node
FQDNs in the talos phase, so a rebuild depends on those records.
The UDM cannot register a network's clients selectively, so the client
goes rather than the record. DHCP is now off on VLAN 47 entirely and
ingress0 declares .11/.12/.13 by node ordinal, matching pod0's
numbering. Its default route is declared with the same losing metric
2048 the lease supplied, so nothing about routing changes: mgmt0 keeps
both default routes until the flip.
IPv4 only. The problem was DHCPv4 announcing a hostname; IPv6 never had
one. The prefix is announced correctly, so ingress0 keeps taking its
IPv6 address by SLAAC exactly as mgmt0 takes its fd40:: address —
derived from the MAC, stable across reboots, nothing to allocate.
accept_ra therefore stays at its default of 2. An IPv6 default router
is a link-local address that can only be learned from the RA, and with
net.ipv6.conf.all.forwarding=1 — which Cilium sets — Linux ignores RAs
below 2 entirely. Only accept_ra_defrtr is set to 0, because mgmt0 owns
the default until the flip; dropping that one line is what later hands
the IPv6 default to ingress0.
pod0 keeps accept_ra 0: it is a closed segment with no router on it, so
there is genuinely nothing to learn.
No reboot needed. Address, route and sysctl changes apply live, and the
whole change is provable per node with `--mode=try`, which reverts
itself.
Refs #576
First pass of the traffic-class split (#576). Purely additive: nothing
moves onto the new segments yet.
Links by role, not by PCI topology. A udev rule per node matches the
burned-in MAC and renames at boot, the way ceph-san already did:
enp2s0f0np0 -> uplink, enp87s0 -> mgmt. Shared manifests now name roles
and only each node's own patch knows the hardware. The two comments
elsewhere that named the old links go with them.
Two new VLANs on the uplink, both created on the UDM already:
- pod0, VLAN 48. Node-to-node only; the overlay rides it once the
kubelet node IP moves. Statically addressed .11/.12/.13 out of
10.2.48.0/24 and fd02:48::/64, no gateway, no DHCP, accept_ra 0. A
lease would make the cluster's data path depend on the UDM being up in
order to form, on a segment whose traffic never reaches the UDM.
- ingress0, VLAN 47. World-to-pod: BGP session, LoadBalancer next hop,
masquerade source. Publicly routed, so DHCP — but at routeMetric 2048
and with accept_ra_defrtr 0, because DHCPv4Config can only
deprioritise a default route, not suppress one, and two defaults at
1024 tie.
mgmt0 keeps the node IP and both default routes. A mistrunked VLAN 47 or
48 therefore cannot strand a node, and the flip is a later change that
needs no reboot and is provable with `--mode=try`.
Applying needs a rolling reboot: a udev rule cannot rename a link on a
running node (Resource busy). Drain first, one node at a time, and wait
for CephCluster HEALTH_OK rather than node Ready between them.
Refs #576
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The tag build-ci-images published from #827 (pipeline 8573): talosctl
v1.13.9, kubectl v1.36.3, flux 2.9.5. Six references moved from
2026-08-10.
Refs #426
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Regenerated with `flux install --export
--components-extra=image-reflector-controller,image-automation-controller`
from the v2.9.5 CLI; the same component set as before.
Controllers: source v1.8.3 -> v1.9.5, kustomize v1.8.4 -> v1.9.5, helm
v1.5.4 -> v1.6.4, notification v1.8.4 -> v1.9.4, image-reflector v1.1.1
-> v1.2.5, image-automation v1.1.2 -> v1.2.5. CRDs updated, the
allow-webhooks NetworkPolicy now names port 9292.
Prerequisites from the 2.9 release notes, verified before this commit:
the removed image v1beta2 and notification v1beta2 API versions are
neither used in the repo (image.* all v1, Alert/Provider v1beta3) nor
stored in the cluster's CRDs (storedVersions all current), so no
`flux migrate` step is needed. helm-controller 1.6 changes the default
post-render strategy; no HelmRelease here uses postRenderers.
`flux check --pre` passes on Kubernetes 1.36.3 and the manifests pass a
server-side dry run against the live cluster.
Refs #426
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
From the update audit (#426): the tools CI drives the cluster with
trailed what the cluster runs.
ci/images/k8s/Containerfile
- talosctl v1.12.6 -> v1.13.9, the Talos version on the nodes. An older
talosctl can misjudge or reject config generated for a newer machine.
- kubectl v1.36.2 -> v1.36.3, the cluster's Kubernetes version.
- flux CLI 2.9.2 -> 2.9.5, the current 2.9.x. Checksums verified against
the upstream release assets (talos sha256sum.txt, flux checksums.txt)
and against a local download.
.woodpecker/ci-manifests-test.yml
- woodpecker-cli v3.17.0 -> v3.18.0, the version the server and agent
already run; the lint step was one minor behind its own server.
Merging triggers build-ci-images (path ci/images/**), which publishes
ci-k8s:<date>. The six pipelines pinned to ci-k8s:2026-08-10 move to
that tag in a follow-up once it exists.
Refs #426
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The description told on-call to try the recovery key as a last resort,
while RUNBOOK Step 4 in the same change (and upstream: recovery keys are
"purely an authorization mechanism") says it cannot unseal an
auto-sealed server. Say what actually helps: fix the Secret, restart
the pod. Also drop a duplicate root-token computation in
restore_openbao.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Follow-up to #825 (static-seal auto-unseal). The rebuild path and the
operator docs still assumed shamir with a manual unseal per replica.
bootstrap.sh
- Required secrets: `seal-key` (pass show homelab/openbao/seal-key)
replaces `unseal-key`. The snapshots are encrypted under the seal key,
so it is the DR-critical secret now; the former unseal key is the
recovery key and is not needed for a restore (upstream: recovery keys
"are purely an authorization mechanism" and cannot unseal a server
whose auto-seal is broken).
- flux phase creates the `openbao-seal-key` Secret (namespace + Secret,
before `flux bootstrap`) from the collected key. --from-literal via a
substitution, because `pass show` appends a newline and OpenBao
rejects a key that is not exactly 32 bytes of base64 ("unknown
encoding for AES-256 key" — verified). An existing Secret with the
same value is left alone; a different value is refused, since the
data is written under whichever key OpenBao started with.
- restore_openbao: `init -recovery-shares=1 -recovery-threshold=1`,
root token from the output, wait for the self-unseal (init returns
with the node sealed; the server's own retry loop unseals it within
~5 s), then `snapshot restore` WITHOUT -force: OpenBao verifies the
snapshot was written under this server's seal key and refuses a
mismatch by name, which is the one guard against restoring under a
wrong Secret. -force skipped that check and would leave a server that
can never unseal. The throwaway `unseal` step and its argv/tty
discussion are gone — under an auto-seal there is no unseal key.
- unseal_openbao -> bring_up_openbao: waits for spec.replicas pods to
report sealed=false, walking the OrderedReady chain as before, and on
timeout prints each still-sealed pod's `bao status` instead of a bare
"timed out".
Rehearsed locally with two OpenBao servers sharing a static key (file://
form, bao 2.5.5): init on an auto-seal returns an empty unseal_keys_b64,
one recovery key and a root token and the server is unsealed within
seconds; a restore of A's snapshot into fresh B (both with and without
force) re-seals and reloads the root key via the auto seal ~5 s later
("done reloading root key using auto seal"), after which B serves A's
data with A's root token. Restarting B under a different key leaves it
sealed with "failed to decrypt keys from storage: message
authentication failed".
Docs and alerts: RUNBOOK-cold-rebuild Step 4 and MIGRATION-sops-openbao
describe the auto-seal flow and the recovery-key semantics; the
OpenBao alert descriptions and the snapshot CronJob comment no longer
tell the reader to unseal by hand. vmtest-job comment updated.
Not covered: a full VMTEST_FULL=1 rehearsal through the flux phase. The
wiki records none ever reaching past the Flux bootstrap checkpoint, so
that gap predates this change.
Refs #805
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every restart sealed the replica until a human unsealed it. On
2026-09-07 a rolling talosctl upgrade rebooted all three nodes; the PDB
refused the second eviction while the first replica was sealed, Talos
waited out its 5-minute drain timeout and rebooted anyway, and raft
quorum was gone for 24 minutes — every ExternalSecret and the
Kustomizations behind them with it. A power loss would do the same
with no timeout to blame.
OpenBao 2.6.1 supports a static seal: a 32-byte key the server uses to
unseal itself. The key is generated into pass (homelab/openbao/seal-key)
and created as the openbao-seal-key Secret before this release reconciles;
the chart hands it to the server as BAO_SEAL_KEY from that Secret.
This overturns the principle in bootstrap.sh that the key protecting
OpenBao must not live in the cluster OpenBao secures. Decided by the
owner on 2026-09-08: the alternatives were a transit seal from a second
OpenBao on edge-1, which raft traffic would wear out, or a workstation
unseal script that may not be at hand when needed. The exposure is
bounded by what ESO already does — every synced secret is plaintext in
etcd (encrypted at rest) — and grows only by what lives solely in
OpenBao.
Migration is a live, per-replica procedure with the shamir key becoming
the recovery key; the StatefulSet is OnDelete, so merging this changes
nothing until each pod is restarted in turn. The bootstrap and rebuild
paths (bootstrap.sh, vmtest) follow in a separate change once the
migration is verified.
Refs #805
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
OpenBaoQuorumLost waited for: 5m. Below quorum the secret store is
down — the ClusterSecretStore goes InvalidProviderConfig and every
ExternalSecret and the Kustomizations behind them stall — and a sealed
replica never recovers on its own, so the window only delayed the page.
It fires on the first evaluation now.
OpenBaoUnavailable, the external probe, drops from 5m to 1m: two
consecutive failed probes, enough to skip a single lost probe through
Traefik, not enough to sit on a sealed store.
Both StatefulSet expressions are wrapped in max by (namespace,
statefulset). kube-state-metrics stamps instance/pod on the series, so
a rescheduled exporter starts a new series and Prometheus resets the
for: timer with it — on 2026-09-07 that made OpenBaoReplicaNotReady fire
14 minutes late. Alertmanager routes match alertname/source only, so the
dropped labels change nothing downstream.
Refs #805
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Grafana dashboard for the gpumon_* series from intel-gpu-exporter:
current Render/3D busy, GPU power and GT frequency per node as stat
tiles, then one time series per engine (Render/3D, Video, VideoEnhance,
Blitter) with node as the series, GT frequency, GPU and package power,
and memory-controller read/write bandwidth.
Node colours are fixed per node across every panel — blue, orange, aqua
for k8s-1/2/3 — rather than assigned by series order, so a node keeps
its colour whatever the $node filter shows. The trio validates for
colour-vision deficiency and normal vision on both Grafana themes
(worst all-pairs ΔE 9.4 / 20.9). Reads and writes share one axis and
differ by dash, not hue. One axis per panel throughout.
gpumon_clients_count is deliberately absent: per-client attribution
needs hostPID, which the exporter does not have, so it would read 0
forever and imply an idle GPU.
Refs #752
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The -device flag named /dev/dri/card0. That is the right node today on
every host, but nothing guarantees the number: a second GPU or a
different probe order and the exporter would silently watch the wrong
device or none. The plugin's monitoring resource injects every GPU on
the node and intel_gpu_top selects it on its own — verified with a probe
pod on k8s-2, which produced the same gpumon_* series without the flag.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A DaemonSet running clambin/intel-gpu-exporter (intel_gpu_top behind a
Prometheus endpoint) with a ServiceMonitor, giving gpumon_engine_usage,
gpumon_frequency, gpumon_power, gpumon_imc_* and gpumon_clients_count
per node.
Privilege model, validated with a one-shot pod before this commit:
/dev/dri comes from the device plugin's gpu.intel.com/monitoring
resource (every GPU on the node, no hostPath); the i915 PMU that
intel_gpu_top reads is a system-wide event source and the nodes run
perf_event_paranoid=3, so CAP_PERFMON is the one capability added.
No privileged, no hostPID, no hostNetwork, read-only root filesystem.
Root rather than a fixed uid because the capability has to be effective
without file capabilities on the binary.
End-to-end check on k8s-2 during a 40 s chat completion on llama-server:
Render/3D busy 98 %, actual frequency 1449 MHz, GPU power 18.8 W, while
the pod's CPU stayed near 100m — the "is it really on the GPU" question
this exists to answer.
XPU Manager was ruled out (Arc Pro / data-centre parts only); the
archived onedr0p exporter needs privileged + hostPID.
Fixes#752
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Enables -enable-monitoring on the device plugin. With 0.36.0 that
advertises gpu.intel.com/monitoring, a single resource per node that
hands a pod every GPU rather than one of the shared slots; it is what a
metrics exporter requests to see the whole iGPU, and it is the only way
to get /dev/dri into a pod without a hostPath.
Nothing consumes it yet. The nodes currently advertise gpu.intel.com/i915
only — the issue assumed the chart default had this on, but the
DaemonSet here is hand-written and did not.
Refs #752
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Both consume their ConfigMap with envFrom, which Kubernetes does not
watch: a changed value applies in git and in the cluster while the
running container keeps the environment it started with. Paperless hit
exactly this in #739 and got the annotation in #741; these two were the
remaining envFrom consumers without one.
ftp-scanner, the third Deployment named in the issue, needs nothing: its
ConfigMap comes from a configMapGenerator that deliberately keeps the
content-hash suffix, so any change renames the ConfigMap, mutates the
pod template and rolls the pod on its own.
Reloader already watches all namespaces (kubernetes/infrastructure/reloader).
Fixes#742
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#817 pinned the digest from the branch build (pipeline 8452). Merging
#816 rebuilt the same date tag on master (pipeline 8457), Gitea kept
only the new manifest, and the pod has been in ImagePullBackOff since:
"failed to resolve reference ...@sha256:0d266b02...". With the Recreate
strategy no controller is running, so the services are unchanged.
The kaniko step of the master build pushed sha256:caffa308...; that is
the digest to pin. Comment states the rule.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Pins the controller image by digest. The registry tag is the build
date and pipeline 8452 rebuilt today's tag in place, so the tag alone
cannot carry a new build to the cluster; the digest can.
Merge only after the Traefik change from 1/2 has reconciled: from this
build on every service target sends a PROXY protocol v2 header, and
Traefik drops such connections unless websecure trusts the pod CIDRs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
TLS-passthrough services hand Traefik a raw TCP stream, so every WAN
request now logs the routing peer's pod IP (10.244.4.64/.112 in the
access log) instead of the client. CrowdSec's detection parses exactly
that log; with the HTTP-mode services gone it would attribute every
external request to two pod IPs, and a re-armed bouncer would repeat the
2026-06-27 mass ban.
The controller now sets options.proxy_protocol on every target (PROXY
protocol v2, TCP/TLS modes only per NetBird) and treats a service
without it as out of date, so the eleven existing services are updated
on the next pass. Traefik's websecure entrypoint accepts the header from
the pod CIDRs, mirroring forwardedHeaders.trustedIPs. Traefik takes
connections with or without a header from a trusted IP, so mesh and LAN
traffic that carries none is unaffected and the Traefik change can land
before the controller image.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First rollout of #814 sat in CreateContainerConfigError: "container has
runAsNonRoot and image has non-numeric user (controller), cannot verify
user is non-root". The kubelet cannot resolve a named user; nobody/nogroup
in the pod spec is enough, the script needs no home and no writable path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deployment for the controller of the previous commit: openbao namespace
like the rotators (its default-deny already allows OpenBao, DNS, the
apiserver and world), own ServiceAccount with a read-only ClusterRole
(Ingresses list/watch, Services get), one replica, Recreate. The image tag
is the branch build that pushed this exact code.
The OpenBao role is applied once by hand (policies/netbird-proxy-services
.hcl); until then the pod logs an auth error per minute and changes nothing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#797 added build-vmtest with gcr.io/kaniko-project/executor:v1.23.2-debug,
the archived Google image the other nine steps left in August (60f231f8).
It passed on master only because that image was still cached on the node;
the branch push of the controller PR scheduled it elsewhere and the step
died before its first command. Same fork pin as the rest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
netbird-proxy-services reconciles the NetBird reverse-proxy services that
nobody managed so far: one TLS-passthrough service on :443 per host of every
Ingress carrying dns.renner.uno/public, targeting Traefik ClusterIP through
a subnet resource over the service CIDR. SNI and Host reach Traefik
untouched, the proxy terminates nothing, no domain resource is involved --
so the dns-interceptor pinning bug that wedges the proxies on every
routing-peer loss has nothing to pin on.
Ownership is derived from NetBird, not from local state: managed iff mode is
tls and the target is the controller subnet resource. The ten hand-made
HTTP/domain services are reported as held and never touched; migration is
deleting one of them, the controller creates the passthrough on its next
pass. The PAT is read from OpenBao per pass through a read-only role
(policies/netbird-proxy-services.hcl, applied once by hand like the rotator
roles). Standard library only; unprivileged image built alongside heating-can.
Validated against the live management (ONCE=1 through the rotator role):
network, resource, router and policy created, 10 hosts held, one host that
had no service at all (llm.renner.uno) created and answering behind its
Authentik gate; second pass 0 created / 0 updated / 0 deleted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every loss of the routing peer a proxy has pinned wedges its route
manager; the flood watchdog restarts the container within 2 s and the
public edge is back after one failed probe round. Inside the 600 s
cooldown the restart is suppressed and the URL watchdog three one-minute
rounds are the only recovery: 3.5 min of 000 on every public domain,
measured 2026-09-07 10:28 on proxy-2 ("restart suppressed, last restart
567s ago"). The morning node roll hit the same suppression on both
proxies at 06:58 and survived only because the routes recovered by luck.
Rolling reboots lose one peer every few minutes by design, so with 600 s
the second roll is always the unlucky one. A minute keeps the loop
protection (one restart per probe interval at most).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Phase 1 (#802) put the ServersTransport under post-crds and
infrastructure-crds has applied it. The copy in crowdsec/lapi-ingress.yaml
goes; the Ingress stays (core kind). With it, the infrastructure apply set
holds no traefik.io consumer any more, which is what a cold rebuild needs.
Prune: infrastructure reconciles first and skips the object if the last
applier label names infrastructure-crds; otherwise it deletes and
infrastructure-crds recreates it seconds later. Only the public LAPI
ingress used by the external bouncers is behind that transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Since the host-device rollout (#801) Rook cannot finish a CephCluster
reconcile: it runs a canary pod with OSD placement to read the cluster
CIDR off the attachment, the OSD on that node already holds the NIC, and
the canary fails "error adding container to network ceph-san" until the
15-minute discovery timeout, then the next reconcile repeats it. Under
macvlan the canary could sit beside the OSD; a whole-device attachment has
exactly one consumer per node.
Discovery is skipped when addressRanges.cluster is set (Rook: "if user
specifies CIDRs in the network config, that is the definitive source of
truth"). The value is what discovery produced and what ceph config already
holds, so no daemon changes. The old comment claiming addressRanges is
honoured only under provider: host was wrong — the CRD documents it for
host and multus.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every node has a second, address-less NIC (enp87s0) up on the untagged
fallback network, which shares the mgmt VLAN's broadcast domain. With
the Linux default arp_ignore=0 the host answers ARP for the node IP from
that NIC too, so the UDM can learn 10.2.40.x against enp87s0's MAC.
Inbound IPv4 then arrives on a device Cilium is not attached to, never
reaches cil_from_netdev, and is routed straight back to the UDM by the
default route — a loop that made every IPv4 LoadBalancer VIP on the
affected node unreachable for 13.5 hours. IPv6 was untouched because
NDP only answers on the interface that owns the address.
The flux was measurable on all three nodes: an ARP request for a node
IP from a neighbour returned replies from two MACs. Which node's entry
had actually flipped at the UDM was a race, and it moved during the
diagnosis.
arp_ignore=1 answers only when the target address is configured on the
receiving interface; arp_announce=2 stops the host from sourcing ARP
from an interface that does not own the address. Applied live with
talosctl on all three nodes before this commit and verified with
enp87s0 up: each node now answers from its mgmt0 MAC only.
A node property, so it lives next to the existing enp87s0 sysctls in
each machine patch rather than in the site patch.
Refs #793, #576
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
All three OSD prepare jobs on the Hetzner rehearsal finished with zero
OSDs:
cephosd: skipping device "sdb1": ["Used by ceph-disk"]
The partition is exactly what the machine config asked for (PARTLABEL
r-ceph-osd, plain Linux partition type). ceph-volume rejects it anyway —
its ceph-disk-member test is `"ceph" in PARTLABEL or PARTTYPE in
ceph_disk_guids` (ceph_volume/util/device.py) — so any label containing
"ceph" reads as a legacy ceph-disk partition and is never provisioned.
Talos derives the label from the RawVolumeConfig name (r-<name>), so the
name changes: ceph-osd -> osd-data, label r-osd-data. This is why the
rebuild branch called it osd-data; the #798 review renamed it back for
symmetry with the node label without this failure being known.
Production is unaffected today: the MS-01 OSDs match on the by-id half of
devicePathFilter and no r-ceph-osd partition exists there. The partlabel
half is the migration target and would have failed on the first
repartitioned drive.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A cold rebuild stops permanently at the infrastructure Kustomization:
ServersTransport/crowdsec/crowdsec-lapi dry-run failed: no matches for
kind "ServersTransport" in version "traefik.io/v1alpha1"
Flux dry-runs the whole apply set, so this one unknown kind rejects all of
it — including the Traefik HelmRelease that installs the CRD. Measured on
the Hetzner rehearsal from master: zero HelmReleases, zero CRDs. The
public LAPI endpoint (2026-09-02) landed after the post-crds sweep and put
a traefik.io consumer back into the layer that creates traefik.io.
Phase 1 only adds the object under post-crds; crowdsec/lapi-ingress.yaml
keeps its copy until infrastructure-crds has applied once, because both
Kustomizations prune and a one-commit move deletes before the new owner
applies. Phase 2 removes the original.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The NetworkAttachmentDefinition switches from a macvlan on a kernel
interface name to host-device on the role name `ceph-san`. The NIC is
dedicated to Ceph and has exactly one consumer per node, so it is moved
into the OSD pod whole — real MAC, real MTU — and returns to the host when
the pod goes away. No hardware name appears in this file any more; every
machine renames its own link at boot (previous commit).
ROLLOUT — not atomic with the machine config. Between this flip and a
node's reboot, an OSD restart on that node fails: the NAD asks for
ceph-san and the running node only has the old name. Merge only inside a
maintenance window, then drain + noout + reboot k8s-1/2/3 back to back.
Running OSD pods keep their existing attachment until restarted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only the Talos machine config knows hardware. Each machine renames its own
Ceph cluster-network link to `ceph-san` at boot, so shared manifests can
address the role and no kernel interface name leaves this directory:
- k8s-1/2/3 (MS-01): a udev rule matches the X710 port 1 by burned-in MAC
and renames it before the network stack opens links. The LinkConfig then
names ceph-san (up, mtu 9000). A rule applied to a RUNNING node cannot
rename (Resource busy) and takes effect at the next reboot, so this
change is inert on production until each node is rebooted — that is the
planned rollout, node by node.
- vm-1..4 (qemu site): one NIC only, so the segment is VLAN 30 named
ceph-san directly. The DHCPv4Config on the parent is load-bearing:
declaring any network document stops the implicit DHCP that addresses
the node.
- hcloud-1/2/3 already carry the same shape (udev by PCI bus path) since
the rebuild-tooling merge.
Validated on Hetzner with the host-device attachment: OSD pods held the
real fabric MACs, Ceph reached HEALTH_OK.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 19.x pin existed to keep this cutover deliberate and tied to the
Rook operator version. That coupling has already come apart in the
dangerous direction: Rook floating constraint pulled v1.20 in June, and
with it cephcsi v3.17.1, whose client libraries are Ceph 20.2.1 — so
every CSI pod has been talking to 19.2.6 mons with a client a whole
major ahead of them.
That is not academic. The 19.2.6 mons mint auth keys as aes256k, and
the 20.2.1 client cannot decode them ("auth: failed to decode key").
Nothing has broken only because the four CSI users predate the cipher
change and still carry legacy aes keys — which Ceph itself flags daily
as HEALTH_WARN (AUTH_INSECURE_CLIENT_KEY_TYPE, 5 entities). The moment
one of those is recreated — ceph auth rotate, a deleted Secret, Rook
resyncing — the provisioner stops and no PVC binds. A cold rebuild hits
it immediately: a fresh cluster has no legacy keys at all.
So the cutover happens now, with the version the client already is.
Rook v1.20 supports Tentacle from v20.2.1; v20.2.0 is excluded
deliberately — upstream reports data corruption there with read
affinity enabled. The two imagepolicy-marked lines are bumped in the
same commit so image automation finds nothing left to do.
Rollout is Rook-managed and rolling: osdMaxUpdatesInParallel: 1 and
upgradeOSDRequiresHealthyPGs: true bound it to one OSD at a time with
clean PGs between steps. Expect mons, then mgrs, then three sequential
OSD restarts with recovery between them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
None of this is reconciled by Flux — bootstrap.sh, the vmtest Job and
the Talos hcloud patches are operator tooling, applied only when a
rebuild test runs. Zero effect on the live cluster from merging.
New: an hcloudtest phase that provisions three cpx42 servers with
private cluster/storage networks, a this-host-only firewall and one raw
OSD volume per node, then hands over to the ordinary talos phase through
the same four variables vmtest uses. Servers carrying the test label are
adopted and rebuilt from the Talos snapshot instead of recreated —
capacity, not cleanliness, is the scarce resource. Network ranges and
volume sizes are derived from what the cluster itself declares (the site
patch and the machine patches), never stated a second time.
hcloudclean deletes everything by label.
The vmtest Job clones from the GitHub mirror instead of in-cluster
Gitea: a rebuild rehearsal cannot depend on the thing whose loss it
rehearses, and a run died on resolving gitea.renner.uno before the
first VM booted.
bootstrap.sh hardening from a week of rehearsals: the OpenBao unseal
loops until the StatefulSet converges instead of unsealing whichever
pods existed at one instant, require_secrets takes a subset, secrets can
be streamed into a waiting run, retries around talosctl bootstrap and
server placement, and node waits print progress and refuse an expected
count of zero.
import-cluster-access.sh keeps its default AWS profile but stops
overriding one the caller already chose.
Known boundary, documented in site-hcloud.yaml: from master the hcloud
site validates provisioning through cluster-Ready; the SAN path needs
the follow-up rollout PR that renames the storage NIC by role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One unknown kind fails a server-side dry-run for the WHOLE apply set,
including the very HelmRelease that would have installed the missing
CRDs. On a running cluster every CRD exists, so this never showed; a
cold rebuild deadlocks on it immediately.
Three groups move, content unchanged:
ExternalSecrets and Traefik Middlewares out of infrastructure into
post-crds, which reconciles after the operators. The flux-git
ExternalSecret also changes creationPolicy Merge -> Orphan with
deletionPolicy Retain: Merge never CREATES the target secret, so on a
fresh cluster the Gitea credential handover silently did not happen.
ServiceMonitors/PodMonitors out of their charts and app sets into
apps/monitoring-rules, which applies after kube-prometheus-stack owns
the CRDs. Rook additionally stops creating its own two monitors
(spec.monitoring.enabled: false) — creating them is fatal to its
node-daemon reconcile when the CRD is missing, which held the
CephCluster in Progressing and with it the whole apps layer. Both
objects are reproduced field for field off the live cluster; the
ceph-exporter DaemonSet and the metrics endpoints are untouched.
Cilium is the worst case: its chart aborts rendering outright on the
missing CRD, and cilium is one of the seven infrastructure
healthChecks — the lowest layer in the cluster waiting on the highest.
Steady-state effect: object ownership moves between Kustomizations;
specs are identical. Verified on local vmtest runs and the Hetzner
throwaway cluster.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gitea runs inside the cluster, so during a cold rebuild every image that
only exists at gitea.renner.uno is unpullable — a rehearsal died with the
cni-plugins installer in ImagePullBackOff, no macvlan binary, no ceph-san
attachment, CephCluster stuck at "Configuring Ceph Mons", OpenBao waiting
forever on a PVC. The failure surfaced four layers from its cause.
The CI images move to konairius/, published by the build pipeline to
Gitea AND ghcr.io with one digest. The manifests keep pointing at Gitea;
the node-level fallback (machine.registries.mirrors, written by
bootstrap.sh at install time) tries Gitea first and ghcr second, and
containerd keeps the repository path when it falls through — so the same
path resolves on both. Steady state never leaves the network.
The Multus, whereabouts and cni-plugins-installer DaemonSets also gain
priorityClassName system-node-critical. Upstream ships them at priority
0: on a rebuild two nodes sat at 99% memory requests when the DaemonSet
rolled, both multus pods stayed Pending, and every pod needing a
secondary attachment waited on an annotation that could never come — with
nothing reporting why.
Rollout effect on the live cluster: the three CNI DaemonSets restart once
(existing pods keep their networking; only new attachments wait), and the
three init containers pull the new tag on their next pod restart. The
ghcr side is proven by the Hetzner throwaway runs, which pull through the
fallback; eyeball gitea.renner.uno/konairius packages before merging.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by rebuild rehearsals; each stopped a cold start outright and none
changes behaviour on a running cluster.
infrastructure-ceph-cluster no longer dependsOn infrastructure. That
edge closed a cycle no new cluster could escape: infrastructure is Ready
only once its openbao healthCheck passes, openbao-0 runs only once its
PVC binds, and the PVC wants the ceph-block StorageClass defined in this
very Kustomization. infrastructure runs wait: false, so everything this
path needs is applied long before readiness; on an empty API server it
self-heals on retryInterval like post-crds already does.
home-assistant gains dependsOn infrastructure-cnpg-barman: its
ObjectStore kind arrives with the Barman plugin, and one unknown kind
fails the server-side dry-run for the whole apply set — taking
speech-to-phrase, zone-configurator and matter-server down with it.
Every other ObjectStore user already declares the edge.
woodpecker-ci namespace moves to infrastructure/namespaces: its
CiliumNetworkPolicies ship with infrastructure-crds, which runs before
every app KS, so declaring the namespace in the workload KS inverted the
order and held infrastructure-crds — and the twenty app Kustomizations
behind it — on "namespaces woodpecker-ci not found". Same reasoning as
its sibling namespace, previously not carried across.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The vmtest Job passes VM_STATE=/work/vm to aim the qcow2 images at the
150Gi emptyDir it sizes for them. bootstrap.sh overwrote it
unconditionally, so the images landed on the container's writable layer
instead and the sizeLimit guard never applied to anything.
Default to the same path as before when the caller sets nothing, so a
plain `bootstrap.sh vmtest` is unchanged.
Two guards come with it, because the value is now caller-controlled:
* a length cap, since the provisioner puts a qemu monitor socket under
this directory and sun_path stops at 108 bytes. Too long used to be a
silent failure to boot; it is now an error. Both the old default (21
bytes) and the Job's /work/vm (8) are well inside it.
* a shape check, because cleanup_vmtest runs `sudo rm -rf "$VM_STATE"`.
That was safe by construction while the path was hardcoded and is not
once a caller picks it.
This does not by itself reduce drive wear — an emptyDir also lives on
/var, so both paths are on the same device. It restores the size guard
and makes it possible to point VM state at a different device later.
VMTEST_FULL and VMTEST_KEEP, which the Job also sets, are still not read
anywhere in this script. Left alone rather than guessed at: the Job is
applied out of band and is not in this repo, so the intended semantics
are not visible from here.
Refs #754
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The evaluation is over. Removing clusters/homelab/paperless-eval.yaml is
the designed teardown: that Kustomization is prune: true with no
prune-disabled annotation anywhere underneath, and its own comment says
being able to delete the namespace by removing the file is the point.
Removed alongside it, so nothing recreates the namespace or points at it:
kubernetes/apps/paperless-eval/ the manifests
clusters/homelab/paperless-eval.yaml the Flux Kustomization
network-policies/paperless-eval-ns.yaml and its kustomization entry
namespaces/namespaces.yaml the Namespace object
network-policies/ai-ns.yaml the llama-embed ingress grant
post-crds/openbao-clustersecretstore.yaml the ESO namespace grant
Production backups are unaffected. The eval Cluster read
s3://renner-uno-k8s-cnpg/pprl-pg as a recovery source with
isWALArchiver: false and no retentionPolicy, so it never wrote to the
production prefix and deleting it removes nothing from the store.
The model weights on the namespace's PVCs were already deleted by hand
before this change; the volumes go with the namespace.
scripts/paperless-eval/ is deliberately left in place — it is the tooling
that drove the evaluation, not part of the namespace, and it is only
useful if the namespace is ever recreated from these manifests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cilium-health reports 0/3 reachable on every node, with Node 1/2 and
Endpoints 2/2 — exactly one of the two node probes fails, including
against localhost.
The Node objects carry only InternalIP=10.2.40.x, because kubelet's
nodeIP.validSubnets lists only the IPv4 mgmt subnet. With no IPv6
InternalIP to take from the Node object, Cilium falls back to detecting
the node's IPv6 from mgmt0 and picks the SLAAC address derived from the
dynamic provider prefix. It records that in CiliumNode and probes it
until the agent restarts, so a prefix change leaves every agent probing
an address that no longer exists:
cilium-health: 2003:d6:9718:3b02:5a47:caff:fe74:7027
mgmt0 now: 2003:d6:9713:b202:5a47:caff:fe74:7027
The interface identifier is the same EUI-64; only the prefix moved. This
is not a one-off — the prefix was 2003:d6:9744:4802 in May and
2003:d6:9718:3b02 in August.
The GUA was never meant to be a control-plane address; that is what the
ULA is for, and BGP already peers on fd40::/64. Listing it in
validSubnets gives the Node a stable IPv6 InternalIP, which Cilium then
uses in preference to its own detection — the same path that already
makes CiliumNode's IPv4 InternalIP agree with the Node object.
Verified fd40::/64 SLAAC addresses are present on mgmt0 on all three
nodes, so no node is left without an address in a listed subnet.
The data path is unaffected either way: Endpoints probes are 2/2 and
conntrack is clean. What this fixes is the health signal, which is
permanently red today and therefore useless for spotting a real outage.
Refs #789
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both socat containers exit 1 on the first datagram they cannot deliver.
While no ready pod backs home-assistant-mdns, its ClusterIP has no
backend and the sendto fails with EHOSTUNREACH, so every Home Assistant
restart costs both containers a kubelet restart — 52 in 10 days.
Run each socat under a shell loop instead. A target that is briefly gone
now costs a two-second gap in what is already a best-effort cache-refresh
path, rather than a restart. Re-running from scratch also re-resolves the
ClusterIP name and rejoins the multicast group.
socat has no option that avoids the wrapper: -s ("sloppy") downgrades the
exit status to 0 but still ends the transfer loop, so the kubelet restarts
the container anyway and the failure stops being visible in the exit code.
The address-level retry/forever options govern opening an address, not
writing to one. Both verified against the pinned image.
The loop traps TERM and forwards it to the backgrounded socat, so the pod
still terminates in ~0.1s; with strategy: Recreate a stalled terminate
would delay every rollout by the full grace period.
Trade-off: a permanently broken relay no longer surfaces as a restart
count. The restart line on stderr is the replacement signal.
Refs #787
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both carried a hand-written exemption in image-automation/README.md. The
exclusion list is for images whose tag format makes a policy impossible;
these two publish clean semver streams and were enrollable all along.
Excluding them on blast-radius judgement left them unowned, and they
drifted.
terminus 0.68.0 -> 0.72.0 (both containers)
trmnl-renderer 0.9.5 -> 0.10.3
That drift is not cosmetic. 0.69-0.72 include "Fixed device provisioner
answer blank API key when device exists", "Fixed Display API image name
to use timestamp instead of checksum" and "Fixed API log create action to
authenticate with API key" — the provisioning, redraw and log paths that
the entrance panel's onboarding was debugged and worked around by hand on
2026-09-05, on a server a month behind the fixes.
The renderer's second stated reason, no upstream multi-arch manifest, is
not an obstruction: the reference already names the architecture, so the
policy pins nothing the tag did not.
Adds apps/trmnl/README.md: which firmware artefact to use for a manual
flash (the mirror carries the C3 build, this panel is S3), what to check
after an image bump, and the device endpoint's scheme constraint.
Watch the panel after this lands: a bad renderer or terminus bump shows
up as a blank or corrupted screen, not as a failed rollout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cleartext LoadBalancer was justified with "the firmware cannot be
handed the internal CA". There is no internal CA in that path:
letsencrypt-cluster solves DNS-01 against the renner.uno Cloudflare zone,
so a LAN-only name gets a publicly-trusted certificate without ever being
publicly reachable — which is why trmnl.renner.uno has always served an
issuer CN=YR2 cert.
trmnl-device.renner.uno therefore moves onto Traefik with its own
Ingress, and API_URI follows it to https so the image_url handed back on
/api/display is fetched over the same scheme the device polls with. The
name moves rather than being duplicated: external-dns runs --policy=sync
and would flap the record between the Traefik VIP and the LB address if
both an Ingress and the Service claimed it.
The LoadBalancer stays, without a hostname. http://10.2.99.89 remains a
path to /api/setup that depends on neither Traefik nor cert renewal, and
the captive portal takes a bare address, so it is the recovery route if
TLS is ever what breaks the panel.
Note Traefik's web entrypoint carries a global http -> websecure
redirect, so http://trmnl-device.renner.uno now answers 308 rather than
serving. The device is provisioned with the https URL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Companion to ai-eval.sh, for the half of the stack that was never measured. The
chat model was chosen from a four-way benchmark; the embedder was taken from a
first recommendation and deployed unexamined. It now backs Paperless's RAG over
a German archive, where retrieval quality decides which past documents the
classifier gets to see.
It cannot work like ai-eval.sh. A chat answer can be read and judged; an
embedding is 1024 opaque numbers. What is measurable is whether the right
document comes back first, so this embeds a labelled German corpus and scores
real questions against it: recall@1, recall@3, MRR, and the cosine margin
between the correct document and the best wrong one — a model that is right by
0.002 is guessing, one that is right by 0.15 has actually separated them.
The corpus is built to be failable. Two Allianz policies, four documents that
are all Rechnungen, a car insurance and a vehicle registration sharing a number
plate. Queries are paraphrases rather than keyword matches — "Wann muss mein
Auto zum TUEV" against a document that only ever says Hauptuntersuchung. An easy
corpus would score every model 100% and rank nothing.
Baseline for the deployed model:
RESULT model=qwen3-embedding-0.6b dim=1024 recall@1=92% recall@3=100% mrr=0.958
The single miss is the intended hard case — the car-insurance question lands on
the household policy, rank 2, margin -0.0132. The margin column also exposes
what pass/fail would hide: arbeitsvertrag is won by +0.0059, a coin flip
recorded as a hit.
Reaches llama-embed from inside the cluster by default, since that Service has
no ingress by design.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 07:28:31 +02:00
292 changed files with 15775 additions and 3503 deletions
MODEL=$$(sh scripts/ci/pick-model.sh "distill raw source notes into atomic wiki notes: read sources, check the wiki for existing coverage, write or merge notes; careful writing, moderate reasoning")
if [ -n "$$MODEL" ]; then echo "distill: selected model $$MODEL"; else echo "distill: using the CLI default model"; fi
cd notes
cat ../ci/agents/wiki-maintenance/distill.md | claude $${MODEL:+--model $$MODEL} \
You are running as the scheduled wiki-maintenance job in CI. The Notes vault is checked out at ~/Notes (the repository k.renner/notes); the homelab and homeassistant repositories are checked out under ~/source/. `kubectl` reads the live cluster with a read-only identity.
Run the wiki-consolidate agent now for today's scheduled consolidation pass. Follow its instructions exactly: snapshot the vault to git first, abort if that fails, select today's rotating cohort of 20 from the state file, verify claims against both their sources and the live systems, act on the verdicts, write the dated report to ~/Notes/output/, update the state file, and snapshot again at the end. Do not push; the pipeline pushes the run's commits.
Report the counts and the coverage progress when you are done.
You are running as the scheduled wiki-maintenance job in CI. The Notes vault is checked out at ~/Notes (the repository k.renner/notes). Do exactly this, nothing else:
Glob ~/Notes/raw/*.md (top level only, never ~/Notes/raw/_processed/). Distill every file you find into ~/Notes/wiki/ following your process, including the ranker check for every concept. Move each fully processed source to ~/Notes/raw/_processed/. If raw/ has no top-level files besides _index.md, say so and stop.
Do not commit or push; the pipeline does that after you finish. Do not touch anything outside ~/Notes/wiki/ and ~/Notes/raw/.
End with one line per file: which wiki notes it produced or merged into, or "nothing distillable".
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Deploy Paperclip (server-local agent runs, option B) into the homelab cluster behind an Authentik gate, with CNPG Postgres and OpenBao-sourced secrets.
**Architecture:** One Flux `Kustomization` (`paperclip`) applies `kubernetes/apps/paperclip/`: a single-replica Deployment on a ceph-block PVC, CNPG `pclp-pg` (PG17) with barman backups to Hetzner, and a Traefik Ingress that backends only the `paperclip-gate` Authentik outpost. Namespace, baseline CNP, ClusterSecretStore allowlist and image automation live in the existing `infrastructure` tree.
- Worktree: `/home/krenner/source/homelab-worktrees/paperclip`, branch `feat/paperclip`. Never edit the main checkout.
- Commits as claude-bot: `git -c user.name='Claude Bot' -c user.email='claude-bot@renner.uno' commit …`, message ends with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
- Push: `git -c credential.helper='store --file=/home/krenner/.config/claude-bot/creds' push …`. PR via `tea pr create --login claude-bot`. Never self-merge.
- One Bash command per call: no `&&`, `;`, `|` chaining, no `2>/dev/null`, no `python -c`.
- Namespace `paperclip`; hostname `paperclip.renner.uno`; LAN-only (no `dns.renner.uno/public` label).
- Image `ghcr.io/paperclipai/paperclip`, tag `nightly`, digest-pinned, `digestReflectionPolicy: Always`.
- Manifest comments: one terse factual line; rationale goes in commit messages/PR.
- No `kubectl apply`/`exec`/`rollout restart` against the live cluster; Flux applies after merge. Server-side `--dry-run=server` is allowed.
## Review Focus
1.**Pod replaced mid agent run** (image bump, secret rotation via Reloader, node drain) — `Recreate` kills in-flight runs; expected: Paperclip marks them failed/recoverable on restart, not wedged. Covered in Task 9 Step 6.
2.**Shared OAuth token expired or missing** — expected: agent runs fail with a visible auth error in Paperclip, server stays healthy. Covered in Task 9 Step 7.
3.**Direct access bypassing the gate** from another namespace pod — expected: connection refused/dropped by CNP. Covered in Task 9 Step 4.
4.**Unauthenticated LAN request** to `paperclip.renner.uno` — expected: redirect to Authentik, never Paperclip's UI. Covered in Task 9 Step 3.
5.**Nightly image with a breaking DB migration** — expected: a base backup exists before the first bump lands. Covered in Task 9 Step 5.
---
### Task 1: Upstream facts that decide two manifest values
No repo changes; produces two decisions consumed by Task 6.
**Interfaces:**
- Produces: `SIGNUP_DISABLED` (`"true"` or `"false"`), `CONFIG_PARTIAL_OK` (yes/no), `ENTRYPOINT_NEEDS_ROOT` (yes/no).
- [ ]**Step 1: Sign-up vs instance claim.** Read upstream `server/src/auth/better-auth.ts`, `server/src/routes/auth.ts`, and the claim flow for `authenticated`+`private` (search `claim` under `server/src`):
Run: `gh api repos/paperclipai/paperclip/contents/server/src/routes/auth.ts -H "Accept: application/vnd.github.raw"`
Decide: if the first-admin claim creates the account through a path that ignores `disableSignUp` (e.g. invite/claim token handler calling the adapter directly), `SIGNUP_DISABLED="true"`. Otherwise `SIGNUP_DISABLED="false"` — the Authentik gate is the access control, and a follow-up PR flips it to `"true"` after the admin is claimed.
- [ ]**Step 2: Partial config.json.** Read `server/src/config.ts` and the config-file loader it calls (search `PAPERCLIP_CONFIG` / `readConfigFile`).
Run: `gh api repos/paperclipai/paperclip/contents/server/src/config.ts -H "Accept: application/vnd.github.raw"`
Decide `CONFIG_PARTIAL_OK=yes` if a file containing only `{"telemetry":{"enabled":false}}` parses (every other section optional/defaulted). If not, stop and report to the user — do not invent config fields.
- [ ]**Step 3: Entrypoint privilege.**
Run: `gh api repos/paperclipai/paperclip/contents/docker-entrypoint.sh -H "Accept: application/vnd.github.raw"`
`ENTRYPOINT_NEEDS_ROOT=yes` if it calls `usermod`/`groupmod`/`chown`/`gosu`. Record the result for Task 6.
- [ ]**Step 4: Anonymous health response.**
Run: `gh api repos/paperclipai/paperclip/contents/server/src/routes/health.ts -H "Accept: application/vnd.github.raw"`
Confirm `GET /api/health` returns HTTP 200 to an anonymous caller when healthy (probes use it). If it returns 401 anonymously, use a `tcpSocket` probe on 3100 in Task 6 instead.
# Traefik is not admitted: the only entry is the paperclip-gate outpost (apps/paperclip/network-policy.yaml).
apiVersion:cilium.io/v2
kind:CiliumNetworkPolicy
metadata:
name:default-deny-ingress
namespace:paperclip
spec:
endpointSelector:{}
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": database
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
- fromEntities:
- host
- remote-node
- fromEndpoints:
- matchExpressions:
- key:io.kubernetes.pod.namespace
operator:Exists
icmps:
- fields:
- {type:0, family:IPv4 }
- {type:3, family:IPv4 }
- {type:8, family:IPv4 }
- {type:11, family:IPv4 }
- {type:12, family:IPv4 }
- {type:1, family:IPv6 }
- {type:2, family:IPv6 }
- {type:3, family:IPv6 }
- {type:4, family:IPv6 }
- {type:128, family:IPv6 }
- {type:129, family:IPv6 }
- {type:133, family:IPv6 }
- {type:134, family:IPv6 }
- {type:135, family:IPv6 }
- {type:136, family:IPv6 }
- {type:143, family:IPv6 }
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperclip
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": authentik
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name:traefik
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": rook-ceph
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
k8s-app:kube-dns
toPorts:
- ports:
- port:"53"
protocol:UDP
- toEntities:
- host
- remote-node
- world
- toEndpoints:
- matchExpressions:
- key:io.kubernetes.pod.namespace
operator:Exists
icmps:
- fields:
- {type:0, family:IPv4 }
- {type:3, family:IPv4 }
- {type:8, family:IPv4 }
- {type:11, family:IPv4 }
- {type:12, family:IPv4 }
- {type:1, family:IPv6 }
- {type:2, family:IPv6 }
- {type:3, family:IPv6 }
- {type:4, family:IPv6 }
- {type:128, family:IPv6 }
- {type:129, family:IPv6 }
- {type:133, family:IPv6 }
- {type:134, family:IPv6 }
- {type:135, family:IPv6 }
- {type:136, family:IPv6 }
- {type:143, family:IPv6 }
```
- [ ]**Step 3: Register it** — in `network-policies/kustomization.yaml` add ` - paperclip-ns.yaml` directly after ` - paperless-ns.yaml`.
- [ ]**Step 4: Allowlist the namespace** — in `openbao-clustersecretstore.yaml` insert ` - paperclip` between ` - monitoring` and ` - paperless` (list is sorted).
Out-of-repo; must exist before merge or the ExternalSecret never becomes Ready.
**Interfaces:**
- Produces: OpenBao KV `apps/paperclip` with the four fields in Global Constraints.
- [ ]**Step 1:** Ask wiki-synth for the claude-bot OpenBao write procedure (note `claude-bot-openbao-write-only-identity`): login method, KV mount, exact `bao kv put`/`patch` syntax for an `apps/<name>` key. Use exactly what it returns.
- [ ]**Step 2: Write the generated fields** as claude-bot (write-only identity — never read values back). Values generated inline by the command:
Generate each with a separate `openssl rand -hex 32` call and pass it directly into the `bao kv put` from Step 1; do not echo or log them.
- [ ]**Step 3: Ask the user** to add the shared token themselves (never handle it): "Run `claude setup-token`, then `bao kv patch <mount>/apps/paperclip CLAUDE_CODE_OAUTH_TOKEN=<token>`." Wait for confirmation.
- [ ]**Step 4: Verify field names only** (no values): `bao kv metadata get <mount>/apps/paperclip` shows a current version ≥2. If claude-bot cannot read metadata, ask the user to confirm the four keys exist.
Expected: both objects `created (server dry run)`; no `unknown field "digestReflectionPolicy"` error.
- [ ]**Step 4: Resolve the current nightly digest.**
The workstation has `docker` only (no skopeo/crane/buildx, checked 2026-09-25); `docker manifest inspect` lists per-platform digests but not the index digest, so pull once:
- Produces: pods labelled `app.kubernetes.io/name: paperclip` listening on 3100; Service `paperclip:3100`; ExternalName `ak-outpost-paperclip-gate:9000`.
- [ ]**Step 1: `pvc.yaml`:**
```yaml
apiVersion:v1
kind:PersistentVolumeClaim
metadata:
name:paperclip-data
namespace:paperclip
annotations:
kustomize.toolkit.fluxcd.io/prune:disabled# data guard
spec:
accessModes:[ReadWriteOnce]
storageClassName:ceph-block
resources:
requests:
storage:20Gi
```
- [ ]**Step 2: `paperclip.yaml`** — replace `<INITIAL_DIGEST>` with the Task 4 value and `<SIGNUP_DISABLED>` with the Task 1 value. If Task 1 found `ENTRYPOINT_NEEDS_ROOT=no`, replace the container `securityContext` with `runAsNonRoot: true`, `runAsUser: 1000`, `runAsGroup: 1000`, `allowPrivilegeEscalation: false`, `capabilities: {drop: ["ALL"]}`. If Task 1 chose `tcpSocket`, replace each `httpGet` block with `tcpSocket: {port: http}`.
Expected: one `pclp-pg-daily-…` with phase `completed`.
- [ ]**Step 6: Image policy.**
Run: `kubectl get imagepolicy -n flux-system paperclip -o jsonpath='{.status.latestRef}'`
Expected: `nightly` tag with a `sha256:` digest.
- [ ]**Step 7: Hand-over to the user (browser steps, theirs):**
1. Open `https://paperclip.renner.uno`, pass Authentik, claim the instance as admin.
2. Create a test agent with the `claude-local` adapter and a trivial task ("reply with OK"); it must complete with the shared token (Review Focus 2: if it fails, report the run's error text — an expired token must show as an auth error, not a hang).
3. If `SIGNUP_DISABLED` was `"false"`, open the follow-up PR flipping `PAPERCLIP_AUTH_DISABLE_SIGN_UP` to `"true"` once the admin is claimed.
4. Review Focus 1 is observed on the first nightly bump: check the interrupted-run status in the UI.
- [ ]**Step 8: Wiki.** Route the verified Paperclip facts (execution topology, env vars, image tags, digest-reflection usage) through wiki-ingest → wiki-distill once the deployment has proven out.
Run [Paperclip](https://github.com/paperclipai/paperclip) (MIT, agent-orchestration
server + UI) in the homelab cluster for real use: its agents do actual work against
homelab systems. This spec covers the **platform** only. Bot identities and their
credentials are out of scope (see "Deferred").
## Decisions
| Topic | Decision |
|---|---|
| Execution model | **B — server-local runs** (upstream default): agent CLIs are child processes of the server, in the server container. Trial only. |
| Upgrade path | If Paperclip is kept: **C** — `@paperclipai/plugin-kubernetes` sandbox provider + kubernetes-sigs agent-sandbox controller (`agents.x-k8s.io`), per-run Pods and Secrets. Separate spec. |
| Claude auth | **One shared Claude subscription AI connection**, created once via Paperclip's local sign-in (`claude auth login` into `ai-local-logins/<id>`); Paperclip stores and refreshes it. Server-env `CLAUDE_CODE_OAUTH_TOKEN` is not used: `stripAiAuthBindings` blanks it for managed runs, and upstream accepts no pasted setup-token (the device-code flow needs a Daytona sandbox). |
| Bot credentials | Out of scope. Future "AIR-Manager" Paperclip role handles permission requests. |
| Exposure | LAN-only, behind an Authentik forward-auth gate, then Paperclip's own login. |
### Known trade-off of B (accepted for the trial)
Upstream `runChildProcess()` spawns every local adapter with
`{...sanitizeInheritedPaperclipEnv(process.env), ...opts.env}`; the sanitizer strips only
`PAPERCLIP_*` keys. Every agent therefore sees `DATABASE_URL`, `BETTER_AUTH_SECRET`,
the shared filesystem (including every other agent's workspace and the
`ai-local-logins` credentials) and every agent-scoped secret it can read from disk. Under B, no bot may hold
a credential we would not tolerate any other bot using. Per-bot isolation is C's job.
(later: Database, Ollama/Onnx, local Jev, pgvector/Qdrant, OpenAI-compatible answer models)
```
Rules that keep it modular, enforced by project references and a test:
-`Wiki.Core` references nothing. It holds the domain types and the port
interfaces. Adapters reference Core only; they never reference each other
or the host. `Wiki.Host` is the only project that knows which adapters
exist.
- Every port has a **contract test suite** in `test/Wiki.Contracts` that
runs against every adapter of that port. A new adapter is done when the
suite is green. That is what makes "swap the storage" a safe operation.
- Adapters are selected by configuration and registered by name (keyed
services), so several can be active at once. The UI's lab page and the
evaluator use that to compare, the API uses the one marked active.
- Nothing in Core or Application knows about HTTP, Gitea, TypeSafe, files or
Kubernetes.
### Ports
| port | what it abstracts | first adapter | examples of the next ones |
|---|---|---|---|
| `INoteStore` | read notes, raw sources and reports at a head; create a new document (a fact becomes a raw note in the application layer, the store only moves text) | Gitea repository API (`k.renner/notes`, blob SHA) | git checkout on a volume; a database (CNPG Postgres) with the vault imported; an Obsidian sync folder |
| `IEmbeddingModel` | text → vector, with a model id and dimension | llama-server `/v1/embeddings` (bge-m3 via wiki-embed, the model that won the German-document comparison on accuracy per cost) | the qwen3 embedding variants from that comparison; ONNX in-process; any OpenAI-compatible endpoint |
| `IVectorIndex` | store vectors per (model id, document), query top-k | in-memory cosine with a file snapshot per model, as today | pgvector in the database; Qdrant |
| `IChunker` | split a document into addressable units for typed questions | paragraph splitter (headings, fences), as in `wiki_ask.py` | sentence windows; semantic chunks |
| `IJudge` | typed questions over a state: Choice, Noul, Score, with probabilities | TypeSafe System One (`jev-latest`, cloud; the wiki records no locally hosted Jev today) | a locally hosted Jev with the same request shape, should TypeSafe offer one; a constrained-output local LLM adapter, flagged as uncalibrated |
| `IAnswerModel` | prose from a question plus evidence: the selected paragraphs with their notes and the judgements; must cite by slug; streaming | Anthropic Messages API (Claude, model from config, Sonnet by default) | any OpenAI-compatible endpoint, including the local llama-server; a different Claude model per question |
| `IEvaluator` | measure a retrieval or judge configuration against the labelled set (questions and cited sources in `output/`) | recall@k, hit rate, MRR, as `wiki_rank_eval.py` | judge accuracy against hand-labelled paragraphs |
| `IIdentity` | who is calling, for attribution of facts and saved answers | Authentik proxy headers (humans), API key (machines), as in shelly-manager | OIDC bearer tokens |
Two ports deserve a sentence of scope each. The judge returns judgements
with probabilities, never prose, and the UI never presents its output as an
answer, only as ranked, quoted evidence. The answer model turns that
evidence into words and is allowed nothing else: it sees the selected
paragraphs and the judgements, not the whole context, and its output is
shown next to the evidence it was given. Both are single model calls inside
the request. The service runs no agent: no tool loop, no vault edits. The
wiki-synth agent in a Claude session remains the path for artifacts and for
multi-question synthesis, and it can call `/questions` itself.
### Solution layout
```
Wiki.slnx
src/
Wiki.Core/ domain types, ports, frontmatter and link parsing
Wiki.Application/ use cases; depends on Core only
Wiki.Adapters.Gitea/ INoteStore (read at a head, create)
| `POST /api/v1/rank` | `{question, k, model?}` → ranked slugs; also served at `/rank` for the existing scripts |
| `POST /api/v1/ask` | `{queries, questions, k, judge?, model?}` → context, typed judgements with the answering paragraphs verbatim, `covered`. For agents and code. |
| `POST /api/v1/questions` | `{question, k?, judge?, model?, answerModel?, save?}` → the answer text with `[[slug]]` citations, the evidence (paragraphs, probabilities), `covered`, model and token usage; streamed as server-sent events when requested; with `save`, the `output/` path and commit |
| `POST /api/v1/facts` | deposit a raw source: title, source, `kind: fact`, observed_at, content, tags → path and commit |
| `GET /api/v1/raw` | the distill queue with ages |
| `GET /api/v1/reports` | `output/` documents by type, newest first |
| `POST /api/v1/lab/eval` | run the evaluator for a (model, judge) pair |
| `POST /webhook` | Gitea push, HMAC as today |
`POST /api/v1/facts` is rejected without `source`, with a `kind` other than
`fact`, or with content under a minimum length. The raw rule is enforced by
shape: there is no field for a hypothesis. The file gets the vault's
provenance frontmatter with `captured_by` set from the API key's identity.
`POST /api/v1/ask` is `wiki_ask.py` moved server-side, so the scripts become
thin clients and agents no longer need a vault checkout. Its result is
judgements and quotations, never prose.
`POST /api/v1/questions` is `/ask` plus one answer-model call, in process:
rank the question, build the context, run the default typed questions
(answered, where, covered), hand the selected paragraphs and judgements to
the answer model, stream the prose back. Below the `covered` threshold it
returns the gap instead of an answer. Time is the sum of the parts, about
0.2 s rank, 1 to 4 s judge, a few seconds of generation. Cost is one model
call over a few thousand tokens, a few cents on Sonnet, against a quarter
dollar for a full agent run.
## Auth and network
- Humans: `wiki.renner.uno`, LAN-only (no `dns.renner.uno/public` label),
Ingress backend `ak-outpost-wiki-gate:9000`, a `wiki-gate-egress` CNP in
`authentik`, the proxy-header handler trusts `X-Authentik-*` only from that
outpost.
- Machines: `wiki-index.renner.uno` with the API-key handler; the workstation
pod as environment variables from Kubernetes Secrets filled by
ExternalSecret from the `openbao` ClusterSecretStore. The pod itself has
no OpenBao access. The Python service fetched its secrets from OpenBao
directly through its own Kubernetes-auth role; that role, its policy and
the `ai` → `openbao` CNP entry are retired with it.
- Egress from the pod: Gitea, wiki-embed, `api.typesafe.ai:443`,
`api.anthropic.com:443`. Every
further adapter that talks to a new endpoint gets its own CNP line, which
is the one place adding an adapter touches the cluster.
## Health endpoints
Three endpoints with three different meanings, wired to the .NET health
checks with tags so that the probes and the humans see the right thing:
| endpoint | meaning | probe |
|---|---|---|
| `GET /health` | the process serves HTTP. Always 200 once Kestrel listens; no dependency checks. | liveness |
| `GET /ready` | the active note store has synced at least once and the active index is loaded; 503 while the first sync or index build runs. A failed later sync keeps the last good state and stays ready. | readiness, plus a startup probe with enough budget for the first sync (the Python service was killed mid-build when its `/health` returned 503 during the initial sync) |
| `GET /api/v1/status` | the full picture for humans and the Settings page: index commit and age, counts, queue lengths, and one entry per registered adapter with its last check result and latency. | none |
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5.5
k.renner
security-bot
issue-fix-bot