4470 Commits
Author SHA1 Message Date
security-bot 1912e6b4a0 chore(image-automation): bump apache/tika:3.3.3.0
Updates managed by image-automation-controller.
2026-10-21 20:45:23 +00:00
security-bot 4d51d99d9e chore(image-automation): bump ghcr.io/immich-app/immich-machine-learning:v3.3.5
Updates managed by image-automation-controller.
2026-10-21 18:52:03 +00:00
security-bot 2ddc736f7e chore(image-automation): bump ghcr.io/immich-app/immich-server:v3.3.5
Updates managed by image-automation-controller.
2026-10-21 18:46:24 +00:00
k.renner 02044f0e4b Merge pull request 'fix(kuma): retry MCB Website 3x at 20s before DOWN' (#1418) from atlas/kuma-mcb-www-retries into main
Reviewed-on: #1418
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-21 12:20:08 +00:00
hermes-bot eed77ecb9a fix(kuma): retry mcb-www three times at 20s before DOWN
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
www.multi-circuit-boards.eu (third-party Apache) occasionally resets a
single TLS handshake with ECONNRESET. Kuma 2.5.6 opens a fresh
connection for every check, so two such resets in a row were enough
for DOWN with retryInterval 60 / maxRetries 1: 14 false episodes since
2026-10-20 21:51 UTC. A 30-min probe from the cluster (559 requests at
3s) saw 557x 200 and 2 handshake resets, and got 200 during Kuma's
DOWN window.

With retryInterval 20 (CRD minimum) and maxRetries 3, DOWN now takes 4
failures in a row over at least 60s, so a real outage longer than
about 1 min still alerts. The other third-party monitors are left
unchanged.

Refs #1407
2026-10-21 11:55:05 +00:00
k.renner c7935b918e Merge pull request 'feat(traefik): move the chart to 42.x, access log on the 41+ schema' (#1411) from atlas/traefik-chart-42 into main
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
Reviewed-on: #1411
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-21 10:48:08 +00:00
security-bot a27af28e2f chore(image-automation): bump gitea.renner.uno/gitea_admin/secret-rotator:2026-10-21
Updates managed by image-automation-controller.
2026-10-21 10:18:44 +00:00
k.renner 70d27f3cdf Merge pull request 'fix(secret-rotator): keep the replaced HA token until the next rotation (#1081, #1084)' (#1414) from alert-fix/issue-1081-rotator into main
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: #1414
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-21 10:18:12 +00:00
k.renner 5aed1f5690 Merge pull request 'feat(monitoring): alert when k8s-gateway stops answering renner.uno Ingress names' (#1415) from fix/monitoring-k8s-gateway-probe into main
Reviewed-on: #1415
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-21 10:17:52 +00:00
k.renner 09d1b3d951 Merge pull request 'chore(ci): bump ci-claude image to 2.1.306' (#1416) from ci-image/claude-2.1.306 into main
Reviewed-on: #1416
2026-10-21 10:17:05 +00:00
k.renner 07e2c98a16 Merge pull request 'feat(hermes): route Gitea PR reviews back to the expert's card' (#1417) from feat/hermes-gitea-pr-review-loop into main
Reviewed-on: #1417
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-21 10:16:55 +00:00
hermes-bot cee1011fbc fix(hermes): review loop follows up review-bot 1803
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
- predates() also guards approvals: an approval older than the card the PR
  line names now (e.g. a follow-up whose expert rebutted without a push)
  no longer completes that card on replay.
- predates() parses created_at as epoch or RFC 3339 and fails loudly when a
  card has none (hermes kanban show --json gives epoch seconds, v0.22.2,
  checked against the CLI), instead of silently never skipping.
- Closing one PR does not complete a card another open PR (stacked stage)
  still names.
- Follow-up cards tell the worker to read every review and comment since the
  previous card: only the newest finding is quoted, older ones that predate
  the card are not added (deliberate, now pinned by a test).
2026-10-21 08:59:58 +00:00
hermes-bot 679d1408e6 fix(hermes): review loop ignores findings handled before the line moved
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot 1802 on #1417 (blocking): marker dedupe was per card, but the PR's
Hermes-Card line moves (follow-up card, coordinator hand-off, gitea_pr.py
adopt), so a replay re-sent findings already handled on the earlier card to
the new one. A finding older than the card the line names now is skipped
(review submitted_at / comment created_at vs the card's created_at); the new
card's worker reads the PR itself.

Also from the review:
- PR merged or closed while its card is in review completes the card
  (hook event pull_request_only, route event pull_request, closed hermes-bot
  PRs of the replay window are replayed); a rebuttal without push otherwise
  left the card in review forever once the owner merged.
- 404 on the PR (system hook fires for repos hermes-bot cannot read) is a
  skip, not an error line per review.
- One route-script run per PR at a time (flock): a delivery and its replay
  can overlap.
- find_review drops dismissed reviews like the replay does.
- Replay checks the run budget before listing; CronJob comment updated.
- Skill: stun-tls-probe.py runs via python3 (ConfigMap mode 0644); rebuttal
  wording says the card waits for the owner.
2026-10-21 08:51:38 +00:00
hermes-bot f77e997dfb feat(hermes): route Gitea PR reviews back to the expert's card
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Experts closed their card with kanban_complete as soon as the PR was open, and
nothing listened to Gitea reviews afterwards (#1412/#1414 REQUEST_CHANGES and
the review-bot suggestions on #1411/#1415 sat unanswered).

Owner decision: the card stays in review until the Gitea review is through; no
follow-up cards for a card that is still open.

- Herkunft: expert PRs carry `Hermes-Card: <card> (<profile>)`. The new helper
  homelab-gitea-pr/scripts/gitea_pr.py sets it deterministically (open/adopt)
  and refuses a second PR for the same branch or `Closes #N` (find/open).
  The card id comes from --card, $HERMES_KANBAN_TASK or the scratch workspace
  name: the terminal tool does not pass HERMES_KANBAN_TASK to commands.
- Events: the system hook adds pull_request_review and pull_request_comment
  (Gitea 1.27.3 routers/api/v1/utils/hook.go; HasEvent maps the approved/
  rejected/comment review types onto pull_request_review). Headers per
  modules/webhook/type.go Event(): pull_request_approved/_rejected/_comment
  for reviews, issue_comment for PR comments; the route lists exactly those.
- Routing in gitea_issues.py (no LLM, [SILENT], HMAC by the gateway as before):
  the review payload carries no review id or commit, so the script reads the
  reviews API and ignores reviews of an older head.
  review-bot APPROVED on the head + card in review -> hermes kanban complete.
  review-bot REQUEST_CHANGES/COMMENT, owner reviews with text, owner PR
  comments -> card in review: hermes kanban reopen-review (back to the
  implementer from review_requested, review quoted as untrusted text); card
  open: comment; card done/missing: follow-up card for the same expert
  (key <owner>/<repo>!<n>@after-<card>, needed because Hermes has no CLI to
  reopen a done card) and the PR's line moves to it; PR without a line:
  coordinator card (key ...@review). A marker per review/comment id makes
  every replay a no-op.
  `hermes kanban request-changes` is not usable here: it needs an active
  reviewer run claimed from review, which only exists with review dispatch.
- kanban.review_dispatch=false on default (Hermes docs: "Set false for
  human-only review boards"); otherwise every card in review would spawn an
  sdlc-review LLM run next to review-bot.
- Replay step in gitea-webhook-sync: review-bot's latest review and owner
  reviews of each open hermes-bot PR's head plus owner PR comments (48 h) are
  re-delivered every 15 min. review-bot can approve before the expert calls
  kanban_request_review; without the replay that card would stay in review.
- homelab-gitea-pr moves from the coordinator's PVC into git (experts never
  had it): review-loop section, one-PR-per-cause rule, the helper. SOUL.md and
  the bridge's card text say: PR cards end with kanban_request_review.
2026-10-21 08:39:45 +00:00
hermes-bot 885e4d723e docs(traefik): name the 41.0.0 break that forced the accessLog move
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Addresses the nit in review-bot review 1795 on 75401220.
2026-10-21 08:28:58 +00:00
hermes-bot aebf78b79a fix(monitoring): k8s-gateway dead-man's switch per record, triage hints
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Addresses review-bot review 1792 on a1b52f5b:
- K8sGatewayProbesAbsent now fires per record (A, AAAA), so one broken
  module no longer hides behind the other's series. New promtool case
  (AAAA-only input) and mutation k8s-gateway-absent-aaaa-dropped.
- K8sGatewayIngressAnswerWrong description names the other cause: the
  auth.renner.uno Ingress (authentik) being gone.
- The Traefik lb-ipam-ips annotation points back at the blackbox modules
  that match the same VIP.
2026-10-21 08:28:36 +00:00
hermes-bot 3445ce7013 Merge remote-tracking branch 'origin/main' into alert-fix/issue-1081-rotator
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Picks up main after #1412 was closed as superseded by this PR.
2026-10-21 08:21:47 +00:00
security-bot 9455d79130 chore(image-automation): bump gitea.renner.uno/gitea_admin/ci-backup-verify:2026-10-21
Updates managed by image-automation-controller.
2026-10-21 08:15:20 +00:00
security-bot 2a60ae3c1b chore(image-automation): bump gitea.renner.uno/gitea_admin/kuma-sync:2026-10-21-0756
Updates managed by image-automation-controller.
2026-10-21 08:15:19 +00:00
security-bot 03ce76eaba chore(image-automation): bump gitea.renner.uno/gitea_admin/review-bot:2026-10-21-0752
Updates managed by image-automation-controller.
2026-10-21 08:15:17 +00:00
security-bot 973a4245d3 chore(image-automation): bump gitea.renner.uno/gitea_admin/authentik-gate-sync:2026-10-21-0755
Updates managed by image-automation-controller.
2026-10-21 08:15:14 +00:00
security-bot 9cea229587 chore(image-automation): bump gitea.renner.uno/konairius/hermes:build.1792569647
Updates managed by image-automation-controller.
2026-10-21 08:05:16 +00:00
hermes-bot 6f1a7b6d96 fix(secret-rotator): never fail the HA run after the store; stage the image rollout
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Folds the good parts of #1412 into this PR and addresses both review-bot
reviews (#1412 review 1789, #1414 review 1791):

- Everything after the OpenBao store is best-effort. Decode, reconnect,
  listing and revokes run in reap_superseded(); reap_summary() turns any
  error into "reap skipped" plus a warning, so rotate_ha_token returns Ok.
  A failed run would start a second pod (backoffLimit: 1) that keeps the
  token just stored and its successor and reaps the token consumers still
  hold until ESO syncs, i.e. the #1081 outage. Two unit tests cover it.
- Warn when the replaced token is not named "<client_name> [<6 hex>]":
  no later run reaps it, so it needs a manual revoke (from #1412).
- Staged image rollout (from #1412): only the three HA CronJobs carry the
  flux-system:secret-rotator marker; the other 17 rotators stay on
  2026-07-30 until a manual HA run validates the new binary, then a
  follow-up adds the marker to them. rotation/kustomization.yaml records
  the exception.
- Merged current main (a6f37e2b, 1691a0ac): the Reloader annotations for
  zone-configurator and speech-to-phrase are already there via #1409.
2026-10-21 07:59:52 +00:00
hermes-bot 754012208c docs(traefik): say why the chart range has no ceiling
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot (review 1787) noted the old explanatory comment was dropped without replacement. The range stays open-ended: the homelab rule is to never pin a Helm chart version, Traefik follows current upstream.
2026-10-21 07:58:59 +00:00
hermes-bot 71fe053af1 Merge remote-tracking branch 'origin/main' into work 2026-10-21 07:58:35 +00:00
claude-bot 4403261c2f chore(ci): bump ci-claude image to 2.1.306
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
2026-10-21 07:53:34 +00:00
hermes-bot a1b52f5b1b feat(monitoring): alert when k8s-gateway stops answering renner.uno Ingress names
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
From 2026-10-08 to 2026-10-16 k8s-gateway ran the chart's default Corefile
(zone example.com, forward . /etc/resolv.conf). Pods stayed Ready, every
renner.uno lookup was forwarded to the UDM and still answered (public names as
CNAME proxy.renner.uno), and nothing alerted. probes-dns.yaml only asks the UDM.

Two blackbox DNS modules ask the k8s-gateway Service (by name, not IP) for
auth.renner.uno A over IPv4 (the path CoreDNS forwards on, 10.96.0.11) and
AAAA over IPv6, and fail unless every answer RR is the Traefik VIP
(10.2.99.86 / fd99::86). A forwarded answer — CNAME or any other address —
fails. CoreDNS is bypassed on purpose: its renner.uno block serves stale for 1h.

K8sGatewayIngressAnswerWrong (critical) fires when the 10m success average is
under 0.8 for 5m: about 7m after a total failure starts, and also when one of
the two replicas is broken (average ~0.5), where a `for` on the raw value would
keep resetting. One or two failed probes stay silent. K8sGatewayProbesAbsent is
the dead-man's switch.

Verified: the module config was run with blackbox_exporter 0.29.0 against the
live cluster resolver (which relays k8s-gateway's answer): A and AAAA succeed;
a k8s-gateway-forwarded name (ha.renner.uno, 10.2.99.87) and a wildcard name
(CNAME proxy.renner.uno) fail. promtool-rules.sh with promtool 3.15.0: 9 suites
green, 41 mutations caught (4 new).

Closes #1290
2026-10-21 07:45:53 +00:00
k.renner 1691a0ac3c Merge pull request 'feat(flux): upgrade components to v2.9.6' (#1410) from atlas/flux-2.9.6 into main
Reviewed-on: #1410
2026-10-21 07:44:09 +00:00
k.renner a6f37e2b68 Merge pull request 'fix(zone-configurator,speech-to-phrase): restart on HA prometheus-token rotation (#1082)' (#1409) from alert-fix/issue-1082 into main
ci/woodpecker/push/ci-claude-refresh Pipeline was successful
ci/woodpecker/push/deploy-claude-workstation Pipeline failed
ci/woodpecker/push/deploy-edge-mqtt-bridge Pipeline was successful
ci/woodpecker/push/deploy-infrastructure Pipeline was successful
ci/woodpecker/push/deploy-netbird-client Pipeline failed
ci/woodpecker/push/deploy-netbird-proxy Pipeline was successful
ci/woodpecker/push/deploy-netbird-server Pipeline was successful
ci/woodpecker/push/deploy-uptime-kuma Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/push/hermes-image Pipeline was successful
Reviewed-on: #1409
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-21 07:42:13 +00:00
hermes-bot 0fd90e07e2 fix(secret-rotator): keep the replaced HA token until the next rotation
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ha-token mode revoked the outgoing LLAT seconds after storing the new one in
OpenBao. Consumers read the k8s Secret, which ESO syncs every 1h, so on
2026-10-01 the ServiceMonitor scrape got 401 from 02:00 to ~02:36Z
(HomeAssistantDown, TargetDown) and zone-configurator crashlooped (#1081,
#1082, #1084).

The reap now keeps the token just stored and the one it replaced, and
revokes only rotator-minted tokens (exact "<client_name> [6 hex]") other
than those two, the same keep-two rule gitea-token mode got in 59e804d9.
The replaced token is revoked by the following quarterly run.

The binary only reaches the CronJobs through a new image, and every
rotator was hand-pinned to secret-rotator:2026-07-30. Enrol the image in
Flux image automation (dated-tag policy, as for ci-backup-verify) and mark
all 20 rotator image lines, so they move together to the newest built tag.
Source between 2026-07-30 and this commit differs only by the Rust 1.91 ->
1.99 build image (68c09786).

Closes #1081
2026-10-21 07:41:16 +00:00
hermes-bot 12c5b30286 feat(traefik): move the chart to 42.x, access log on the 41+ schema
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Chart 41.0.0 renamed `logs.access` to `accessLog` and its values schema
rejects the old root `logs` key; that is why 399f101b capped the range
below 41.0.0. Moving `logs.access.enabled` to `accessLog.enabled` makes
our values valid on the current line, so the cap goes.

Verified with helm v4.3.1 against the published charts, values taken
from this HelmRelease, --kube-version 1.36.3:
- 42.0.0 with the old values fails exactly as in June
  ("additional properties 'logs' not allowed").
- 42.1.0 with these values renders the same manifests as the running
  40.3.0, apart from chart/version labels and the image
  (traefik v3.7.4 -> v3.7.15). The access-log args are unchanged
  (--accesslog=true, fields defaultmode keep, headers drop), so the
  CrowdSec agents parsing Traefik logs see the same format.
- Chart CRDs: same 25 CRDs; additive schema changes and controller-gen
  0.17.1 -> 0.21.0 annotations.

Other breaking notes for 41/42 do not apply here: providers.file.content
is not set; 42.0 requires Kubernetes >= 1.30 (cluster runs 1.36.3);
the Hub secret RBAC and transparency-log witness changes only apply
with hub.token, which is not set.

The range is now open-ended (>=42.0.0), per the house rule not to pin
chart versions; note the wiki's floating-helm-chart-constraint-risk
recommends tilde ranges instead, so the owner should confirm the shape.

Refs #426
2026-10-21 07:35:25 +00:00
hermes-bot 947200e081 feat(flux): upgrade components to v2.9.6
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Regenerated with `flux install --export
--components-extra=image-reflector-controller,image-automation-controller`
from the v2.9.6 CLI (release tarball, sha256 checked against
flux_2.9.6_checksums.txt); the same component set as before.

Controllers: source v1.9.5 -> v1.9.6, kustomize v1.9.5 -> v1.9.6, helm
v1.6.4 -> v1.6.5. notification, image-reflector and image-automation
are unchanged. No CRD, RBAC or NetworkPolicy changes: the diff is the
three images and their app.kubernetes.io/version labels.

`flux check --pre` (v2.9.6 CLI) passes on Kubernetes 1.36.3.

Flux core: merge only 02:00-05:00 Europe/Berlin.

Refs #426
2026-10-21 07:34:24 +00:00
hermes-bot c0a3d2e974 fix(zone-configurator,speech-to-phrase): shorten the Reloader annotation comments to one line
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
2026-10-21 07:33:45 +00:00
hermes-bot 38fa4e9df9 Merge remote-tracking branch 'origin/main' into alert-fix/issue-1082 2026-10-21 07:33:33 +00:00
k.renner 2dbd53fb29 Merge pull request 'feat(hermes): Gitea issues become kanban cards; alert issues end to end, issue-fix retired' (#1408) from feat/hermes-gitea-issues-kanban into main
Reviewed-on: #1408
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-21 07:24:45 +00:00
hermes-bot 3aa78c1d0f fix(hermes): put the expert-card guard above the quoted issue text
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot on 5de84e94: the guard sat after up to 20000 characters of untrusted
issue text. render() now takes a head block placed right after the title line;
Gitea-Issue: stays the last line.

Also checked (Gitea release/v1.27 routers/api/v1/repo/collaborators.go,
GetRepoPermissions; the API docs only list 200/403/404): for a site-admin caller
it answers 200 with the access mode (none for a non-collaborator), 404 only for an
unknown user, so repos hermes-bot cannot see are filtered, not failed.
2026-10-21 06:36:55 +00:00
hermes-bot 5de84e9471 fix(hermes): guard the hand-over claim window; skip repos whose permission lookup fails
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot on b6179a41:
- Expert card bodies open with "This card is for the <expert> expert. If you are
  not <expert>, do no work on it": a coordinator that claims the card between the
  body edit and the reassign completes it instead of running the alert protocol
  next to the separate @expert card.
- issue_repos skips (and names) a repo whose hermes-bot permission lookup fails
  instead of aborting; the other repos still get labels and backfill, the Job
  still fails.
2026-10-21 06:30:10 +00:00
hermes-bot b6179a418a fix(hermes): keep the Gitea admin PAT out of the agent's OpenBao reach; safer hand-over
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot on de080442:
- The site-admin PAT moves from apps/hermes/gitea-admin (readable by the agent's
  `hermes` role, homelab/data/apps/hermes/*) to apps/gitea-webhook-sync/admin.
  Only ESO reads it (eso role, existing ClusterSecretStore); `bao token
  capabilities` with the hermes role: deny on the new path. ExternalSecret and
  K8s Secret are now gitea-webhook-sync-admin, mounted only by the CronJob.
- Hand-over writes the expert body before reassigning. If the coordinator claims
  the card in between, its routing body is restored and the expert gets the
  separate @expert card; if the edit fails, nothing is reassigned. Tests for both.
- [alert] cards: a comment is the owner's only by its API user.login (k.renner,
  the login issue-fix trusted and the one commenting on homelab issues), never by
  a name or quote in the text.
- Labels and backfill only in repos where hermes-bot has issue write (per
  /collaborators/hermes-bot/permission), org labels only for orgs owning such a
  repo; gitea_admin/_cargo-index (hermes-bot: read) no longer gets labels.
- Backfill starts no replay after RUN_BUDGET_SECONDS (900); activeDeadlineSeconds
  1200 covers that plus one 60 s delivery; a test pins the relation.

Verified live: Gitea 1.27.3 swagger says IssueLabelsOption.labels takes IDs or
names, so the coordinator's {"labels": ["<expert>"]} works; hermes-bot has
write on konairius/homelab.
2026-10-21 06:23:36 +00:00
hermes-bot de080442d3 feat(hermes): alert issues end to end: labelled at filing, routed, fixed by the expert card; retire issue-fix
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
The owner wants the path alert -> [alert] issue -> kanban card -> expert fix PR
-> owner merges, without a human step in between. Owner decisions (kanban card
t_3f5d0a62): [alert] issues are not excluded from the bridge, and the issue-fix
Woodpecker pipeline is retired, so the expert's card is the only automated fix
path for [alert] issues.

Routing
- alert-monitor: `alerts.py file` takes a required `--label` (hestia, atlas,
  apollo, clio, daedalus) and sets that label at creation (POST /issues
  `labels: [id]`). Gitea 1.27.3 adds the labels in the same transaction
  (services/issue/issue.go NewIssue) and the `opened` webhook payload loads them
  (services/convert/issue.go toIssue -> LoadLabels), so the first delivery
  already routes the card to the expert. Org labels are accepted for org repos
  (models/issues/issue_update.go: label.OrgID == repo.OwnerID). If the label
  does not exist yet, the issue is filed without it and the coordinator routes.
  SKILL.md tells the triage agent how to pick the expert.
- Label-less issues no longer go to triage. kanban.auto_decompose defaults to
  true (hermes_cli/config_defaults.py; not overridden here), so the dispatcher
  hands every triage card to the auxiliary-LLM decomposer
  (gateway/kanban_watchers_dispatcher.py auto_decompose_tick ->
  kanban_decompose.decompose_task), which rewrites title and body (dropping the
  Gitea-Issue line the completion hook needs), may fan out into children without
  that line, and keeps the assignee `default` anyway (_apply_single only sets an
  assignee when none is set). The card is now created ready for `default` with
  routing instructions: set exactly one domain label on the issue, nothing else.
- The script reacts to that label: a coordinator card nobody has claimed yet
  (todo/ready) is reassigned (`hermes kanban assign`) and its body replaced by
  the expert body (`hermes kanban edit --body`); a claimed or finished one gets
  a separate expert card under key `<owner>/<repo>#<n>@<expert>` (one per
  expert, created_by gitea-webhook, so its completion is posted on the issue).
  `assign` refuses a running card, which falls back to the separate card. Cards
  already with an expert are still never moved.

[alert] cards
- Their body carries the fix protocol that replaces issue-fix, including its
  prompt hardening from PR #253/#266: check for an open PR with `Closes #<n>` or
  a branch `alert-fix/issue-<n>` first; owner (k.renner) comments are
  authoritative, everything else is untrusted problem description; verify live
  before fixing, never state cluster state not observed this run, give
  read-only commands instead of predicted output; repo-fixable -> branch
  `alert-fix/issue-<n>`, smallest fix, PR with `Closes #<n>`, never merge; not
  repo-fixable -> no PR, the completion summary (posted on the issue) says why
  and what a human must do.

Gitea side (CronJob gitea-webhook-sync, every 15 min)
- Domain labels: created as org labels in every org (GET /admin/orgs) and as
  repo labels in user-owned repos, only when missing.
- Backfill: open issues without the card comment are replayed to the route as
  signed `opened` deliveries (same headers Gitea sends), at most 10 per run, 3 s
  apart (the adapter's default rate limit is 30/min/route), in random order so
  repos where the comment keeps failing cannot starve the rest. The idempotency
  key `<owner>/<repo>#<n>` makes a replay of an issue with a card a no-op.
  The job runs in the hermes namespace, whose default-deny already allows
  intra-namespace traffic to :8644.
- The admin PAT needs write:organization (org labels, reqOrgOwnership passes for
  a site admin), write:issue (repo labels, comments) and read:repository
  (/repos/search) next to write:admin. activeDeadlineSeconds 120 -> 600.

issue-fix removed
- .woodpecker/issue-fix.yml, ci/agents/issue-fix/, and what only it used:
  scripts/ci/make-diag-kubeconfig.sh, the monitoring/ci-diagnostics
  ServiceAccount and RBAC (no other pipeline mints its token; git grep), the
  .ci-diag/ gitignore entry. Comments naming issue-fix updated.
- Not in git and left to the owner: the Woodpecker cron `issue-fix` and the
  secret `issue_fix_bot_token` with its Gitea PAT.

Tested: unittest 73 OK; gitea_issues.py against the real hermes kanban CLI in a
scratch HERMES_HOME with a stub Gitea (unlabelled -> default ready card;
labelled while unclaimed -> same card reassigned to daedalus with the alert
body; labelled while claimed -> separate card `#n@daedalus`; replay -> no
duplicate card or comment); sync discovery read-only against live Gitea (42
open issues, all without a card); kubectl kustomize for apps/hermes,
apps/monitoring, apps/monitoring-rules.
2026-10-21 05:48:01 +00:00
hermes-bot 82f5b3e95d feat(hermes): Gitea issues become kanban cards
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Every open issue on gitea.renner.uno lands on the Hermes kanban board without a
manual step (owner decision: all issues, no opt-in label). Exactly one label out
of hestia/atlas/apollo/clio/daedalus assigns the card to that expert; anything
else goes to the coordinator (default) in triage.

Inbound, event-driven through upstream Hermes features only (webhook platform,
route script, kanban CLI; nothing forked or patched):

- A Gitea system webhook (events issues_only + issue_label) posts to
  hermes.hermes.svc.cluster.local:8644/webhooks/gitea-issues. Gitea signs with
  X-Hub-Signature-256 and sends X-GitHub-Event: issues, which the Hermes webhook
  adapter validates as GitHub-style HMAC.
- Route gitea-issues runs skills/gitea-issues/scripts/gitea_issues.py as its
  route script. The script creates the card itself with
  `hermes kanban create --idempotency-key <owner>/<repo>#<number>
  --created-by gitea-webhook` and answers [SILENT], so no agent run and no LLM
  call happens. Edits and label changes reuse the existing card (no duplicate,
  no reassignment). The issue text is quoted into the card body as untrusted
  input; the card ends with a `Gitea-Issue: <url>` line.
- It then comments the card id on the issue once (marker comment, so retries
  and later events do not repeat it).

Back channel: a kanban_task_completed shell hook (config `hooks:` on default and
all five experts) runs the same script in `completed` mode and posts the
completion summary on the issue. Dispatcher-spawned workers run with
--accept-hooks; the gateway (dashboard completions) gets an allowlist entry in
each profile's shell-hooks-allowlist.json. The issue is not closed
automatically: a finished card is not proof the issue is resolved (the work may
sit in an unmerged PR), so closing stays with the owner or an explicit ask.

Gitea side, declarative: CronJob gitea-webhook-sync (hermes ns, every 15 min,
hermes image, scripts/sync_webhook.py) keeps the system hook in place through
the admin API. Gitea 1.27.3 facts it relies on (routers/api/v1/utils/hook.go):
POST /admin/hooks creates a *default* (template) hook unless
config.is_system_webhook is "true", and PATCH ignores config.secret. So the
hook is named hermes-kanban-<hash of url, events, secret>; when no active hook
has the desired name, a new one is created first and older hermes-kanban* hooks
are deleted afterwards. Other system hooks (review-bot) are never touched.

Secrets (OpenBao, apps/hermes only, bootstrap commands in
gitea-webhook-externalsecret.yaml):
- apps/hermes/gitea-webhook#secret: HMAC secret, read by the gateway (written
  to /opt/data/.env as WEBHOOK_SECRET by default-env, which also sets
  WEBHOOK_ENABLED only when the secret exists) and by the sync job.
- apps/hermes/gitea-admin#api-token: site-admin PAT (write:admin; Gitea has no
  narrower scope for system hooks), mounted only into the sync job.
The Deployment carries a Reloader annotation for hermes-gitea-webhook, so the
first sync of the secret (and a rotation) restarts the gateway with it.

Network: Service port webhook 8644, CNP hermes-allow-gitea (ingress from the
gitea pod on 8644), egress rule in gitea's default-deny, and
hermes.hermes.svc.cluster.local in GITEA__security__ALLOWED_HOST_LIST.

Tested offline: tests/test_gitea_issues.py (routing, card shape, marker
spoofing, comment dedup, completion, sync create/no-op/replace). Against the
real Hermes code in a scratch HERMES_HOME with a stub Gitea API: WebhookAdapter
accepted a Gitea-style signed POST (200, script), rejected a bad signature
(401), ignored a pull_request event; repeated issue events produced one card;
the init-container logic set the route, hooks and allowlist (`hooks list`:
allowed); kanban_db.complete_task after register_from_config fired the hook and
posted the summary.
2026-10-21 05:29:09 +00:00
security-bot 0a4ef9deae chore(image-automation): bump lscr.io/linuxserver/lidarr:3.1.0.4875-ls45
ci/woodpecker/cron/issue-fix Pipeline was successful
Updates managed by image-automation-controller.
2026-10-21 00:12:50 +00:00
k.renner 0c7e0c0485 Merge pull request 'fix(hermes): write default's platform credentials to .env (Matrix/API down since #1405)' (#1406) from fix/hermes-multiplex-env into main
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: #1406
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-20 20:51:08 +00:00
claude-bot a955cb9f9d fix(hermes): write default's platform credentials to .env (Matrix/API down since #1405)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Since #1405 the pod has six profiles, so the gateway multiplexes
(gateway.multiplex_profiles is written true on boot). In that mode every
profile, default included, reads its platform credentials only from its
own secret scope (/opt/data/.env plus secret sources), never from the pod
environment. After the #1405 rollout the gateway logged "matrix is enabled
but no profile ... provided a bot credential" and started neither Matrix
nor the API server (API_SERVER_KEY is not a global env var). The
GATEWAY_MULTIPLEX_PROFILES=false override does not avoid this: tested in
build.1792397861 with five profiles present, the API server still stays
down.

New init container `default-env` writes API_SERVER_KEY, the MATRIX_*
settings, MATRIX_ACCESS_TOKEN and HA_TOKEN into /opt/data/.env on every
start, through Hermes's own writer (hermes_cli.config.save_env_value
replaces an existing line in place and keeps the 0600 mode). A key whose
source is gone (for example the optional hermes-homeassistant Secret) is
removed from .env with remove_env_value, so a revoked token does not
linger. HA_TOKEN is written too, so default's ${HA_TOKEN} MCP header
resolves in its own scope.

Unlike the other init containers this one is fatal: if a Hermes bump
moves the writer, the pod should fail visibly (HermesDown) rather than
come up Ready with Matrix and the API server silently down.

The plain MATRIX_* settings move into a generated ConfigMap,
hermes-matrix-config, used by envFrom in both containers, so they exist
once. The hermes container keeps its env otherwise unchanged: s6 stage2
checks API_SERVER_KEY there, and expert kanban workers resolve HA_TOKEN
from it. A new profile does not copy the root .env (checked:
`hermes profile create` with credentials in the root .env leaves the new
profile's .env without them), so experts still get no platform
credentials.

Trade-off: these credentials now also sit in the PVC's .env (0600, next
to the keys entered in the dashboard) and in any PVC backup. The same uid
can already read them from the gateway's /proc/<pid>/environ, so this is
no new boundary, but the file is easier to stumble into.

Verified locally in build.1792397861 on a volume with extra profiles and
gateway.multiplex_profiles: true: the script exits 0 on repeated runs,
each key occurs once in .env, an unset HA_TOKEN is removed on the next
run, an unwritable HERMES_HOME exits 1, and the gateway then starts the API server
("API server listening on http://0.0.0.0:8642") and connects Matrix with
the token from .env (the test homeserver was unreachable on purpose).

After merge: check the gateway log for "matrix connected" and
"api_server connected", and that https://hermes-api.renner.uno answers.
2026-10-20 22:48:24 +02:00
k.renner 507bacc5fe Merge pull request 'feat(hermes): five domain experts as profiles, default hands them kanban cards' (#1405) from feat/hermes-experts into main
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: #1405
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-20 20:27:01 +00:00
claude-bot bb3cee8a3c feat(hermes): five domain experts as profiles, default hands them kanban cards
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Adds five expert profiles to the existing hermes pod, managed from git:
hestia (home automation), atlas (platform), apollo (media), clio
(documents and web), daedalus (dev and CI). Their main purpose is to be
used by the default profile, which acts as coordinator.

- experts/<name>.md: each expert's role and domain (apps, repos, domain
  rules). ConfigMap hermes-experts.
- New non-fatal init container `experts`: creates each profile once
  (`hermes profile create`), then on every start writes its SOUL.md
  (shared SOUL.md + role file), its description (the kanban routing hint),
  the Bot Mode title in profile.yaml (`ui_meta.hermes-bots`, which also
  enables message_agent in Bot Chats), skills.external_dirs, and links the
  hermes-bot tea login into the profile HOME (container profiles get
  HOME=<profile>/home), and copies default's Home Assistant MCP entry
  (#1404) into each expert, read from the root config.yaml so there is one
  source; ${HA_TOKEN} resolves from the pod env in expert workers,
  verified. It also enables the kanban toolset for
  default on matrix, cli and api_server.
- SOUL.md: the HA MCP server added to "What you can touch" (#1404 left
  it out; every profile now has it). "The team" section, shared by all profiles: the roster; the
  coordinator hands in-domain work out as self-contained kanban cards and
  relays the outcome; experts stay in their domain and finish with
  kanban_complete / kanban_block.

- README: the HA-impact reviewer stays the owner until the owner hands
  that review to the new `hestia` profile.

How it runs: the kanban dispatcher already runs inside the default gateway
and spawns `hermes -p <expert>` workers for cards. Experts have no
messaging platforms of their own (secondary profiles read platform
credentials only from their own .env), so nothing double-binds Matrix or
the API server. They use the root Anthropic OAuth login; refreshes are
written back to root.

Same access as default, by owner decision: read-only kubectl/talosctl/
bao, PRs as hermes-bot, Home Assistant through ha-updates and the HA MCP
server. HA MCP write tools stay behind trust: untrusted; kanban workers
run as `hermes chat -q`, where the trust gate never auto-accepts
(tools/approval_prompt.py request_elicitation_consent: single-query
sessions decline, gateway prompts nobody answers end as cancel). All
profiles share the pod and its mounted secrets, so this is not a
credential boundary.

Verified locally in the current image (build.1792397861) on a fresh
volume: the init script runs clean twice (idempotent) and produces the
profiles with the composed SOUL.md, title, description, external_dirs
and tea link; the gateway boots with all six profiles; a kanban card
assigned to hestia is claimed and spawned as a worker, and that worker
connects to its MCP server with ${HA_TOKEN} resolved from the env.

Trade-off: `hermes tools enable kanban --platform X` snapshots the current
default toolset list for that platform into config.yaml, so toolsets
Hermes adds by default later will not appear on matrix/cli/api_server
without another enable.

After merge: nothing manual. Check `hermes profile list` and the experts
init log in the new pod, then give default one domain question on Matrix.
2026-10-20 22:19:19 +02:00
security-bot ac44e7a1ff chore(image-automation): bump lscr.io/linuxserver/lidarr:3.1.0.4875-ls44
Updates managed by image-automation-controller.
2026-10-20 20:12:42 +00:00
security-bot 820ca25ea3 chore(image-automation): bump gitea.renner.uno/gitea_admin/kuma-sync:2026-10-20-1950
Updates managed by image-automation-controller.
2026-10-20 19:56:45 +00:00
k.renner 2007cbe7fd Merge pull request 'feat(hermes): add Home Assistant MCP server' (#1404) from feat/hermes-ha-mcp into main
Reviewed-on: #1404
2026-10-20 19:54:25 +00:00
k.renner 8332bfb166 Merge pull request 'feat(kuma-sync): declare the public status page in git' (#1403) from feat/kuma-status-page into main
ci/woodpecker/push/deploy-uptime-kuma Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: #1403
2026-10-20 19:49:51 +00:00
hermes-bot d9e47807bd feat(hermes): add Home Assistant MCP server
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Registers https://home.renner.uno/api/mcp as mcp_servers.homeassistant via
an init container (same pattern as the other config set steps), so the
entry is reapplied on every start instead of living only on the PVC.

Auth reuses Hermes's own HA admin token (apps/hermes/homeassistant, already
synced to hermes-homeassistant for ha-updates), exposed as HA_TOKEN and
referenced as ${HA_TOKEN} in the header; Hermes interpolates it at connect
time, so the token never lands in config.yaml.

trust: untrusted makes every tool without readOnlyHint (HassTurnOn, scripts,
media, vacuum, ...) require approval; read-only tools (GetLiveContext,
calendar/todo reads) run freely. This keeps the standing "no agent write
calls to the HA API" decision enforced per call instead of silently lifted.
2026-10-20 19:48:25 +00:00