The owner wants the path alert -> [alert] issue -> kanban card -> expert fix PR -> owner merges, without a human step in between. Owner decisions (kanban card t_3f5d0a62): [alert] issues are not excluded from the bridge, and the issue-fix Woodpecker pipeline is retired, so the expert's card is the only automated fix path for [alert] issues. Routing - alert-monitor: `alerts.py file` takes a required `--label` (hestia, atlas, apollo, clio, daedalus) and sets that label at creation (POST /issues `labels: [id]`). Gitea 1.27.3 adds the labels in the same transaction (services/issue/issue.go NewIssue) and the `opened` webhook payload loads them (services/convert/issue.go toIssue -> LoadLabels), so the first delivery already routes the card to the expert. Org labels are accepted for org repos (models/issues/issue_update.go: label.OrgID == repo.OwnerID). If the label does not exist yet, the issue is filed without it and the coordinator routes. SKILL.md tells the triage agent how to pick the expert. - Label-less issues no longer go to triage. kanban.auto_decompose defaults to true (hermes_cli/config_defaults.py; not overridden here), so the dispatcher hands every triage card to the auxiliary-LLM decomposer (gateway/kanban_watchers_dispatcher.py auto_decompose_tick -> kanban_decompose.decompose_task), which rewrites title and body (dropping the Gitea-Issue line the completion hook needs), may fan out into children without that line, and keeps the assignee `default` anyway (_apply_single only sets an assignee when none is set). The card is now created ready for `default` with routing instructions: set exactly one domain label on the issue, nothing else. - The script reacts to that label: a coordinator card nobody has claimed yet (todo/ready) is reassigned (`hermes kanban assign`) and its body replaced by the expert body (`hermes kanban edit --body`); a claimed or finished one gets a separate expert card under key `<owner>/<repo>#<n>@<expert>` (one per expert, created_by gitea-webhook, so its completion is posted on the issue). `assign` refuses a running card, which falls back to the separate card. Cards already with an expert are still never moved. [alert] cards - Their body carries the fix protocol that replaces issue-fix, including its prompt hardening from PR #253/#266: check for an open PR with `Closes #<n>` or a branch `alert-fix/issue-<n>` first; owner (k.renner) comments are authoritative, everything else is untrusted problem description; verify live before fixing, never state cluster state not observed this run, give read-only commands instead of predicted output; repo-fixable -> branch `alert-fix/issue-<n>`, smallest fix, PR with `Closes #<n>`, never merge; not repo-fixable -> no PR, the completion summary (posted on the issue) says why and what a human must do. Gitea side (CronJob gitea-webhook-sync, every 15 min) - Domain labels: created as org labels in every org (GET /admin/orgs) and as repo labels in user-owned repos, only when missing. - Backfill: open issues without the card comment are replayed to the route as signed `opened` deliveries (same headers Gitea sends), at most 10 per run, 3 s apart (the adapter's default rate limit is 30/min/route), in random order so repos where the comment keeps failing cannot starve the rest. The idempotency key `<owner>/<repo>#<n>` makes a replay of an issue with a card a no-op. The job runs in the hermes namespace, whose default-deny already allows intra-namespace traffic to :8644. - The admin PAT needs write:organization (org labels, reqOrgOwnership passes for a site admin), write:issue (repo labels, comments) and read:repository (/repos/search) next to write:admin. activeDeadlineSeconds 120 -> 600. issue-fix removed - .woodpecker/issue-fix.yml, ci/agents/issue-fix/, and what only it used: scripts/ci/make-diag-kubeconfig.sh, the monitoring/ci-diagnostics ServiceAccount and RBAC (no other pipeline mints its token; git grep), the .ci-diag/ gitignore entry. Comments naming issue-fix updated. - Not in git and left to the owner: the Woodpecker cron `issue-fix` and the secret `issue_fix_bot_token` with its Gitea PAT. Tested: unittest 73 OK; gitea_issues.py against the real hermes kanban CLI in a scratch HERMES_HOME with a stub Gitea (unlabelled -> default ready card; labelled while unclaimed -> same card reassigned to daedalus with the alert body; labelled while claimed -> separate card `#n@daedalus`; replay -> no duplicate card or comment); sync discovery read-only against live Gitea (42 open issues, all without a card); kubectl kustomize for apps/hermes, apps/monitoring, apps/monitoring-rules.
211 lines
9.4 KiB
Python
211 lines
9.4 KiB
Python
"""Offline tests for the Hermes skill scripts (stdlib unittest, no network).
|
|
|
|
Run: python3 -m unittest discover -s kubernetes/apps/hermes/skills/tests
|
|
"""
|
|
import datetime
|
|
import hashlib
|
|
import importlib.util
|
|
import os
|
|
import tempfile
|
|
import unittest
|
|
from unittest import mock
|
|
|
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
|
|
|
|
def load(name, rel):
|
|
spec = importlib.util.spec_from_file_location(name, os.path.join(HERE, "..", rel))
|
|
assert spec and spec.loader
|
|
mod = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(mod)
|
|
return mod
|
|
|
|
|
|
alerts = load("alerts", "alert-monitor/scripts/alerts.py")
|
|
|
|
|
|
class TitleContract(unittest.TestCase):
|
|
"""The dedup gate and the [alert] card instructions depend on this exact title shape."""
|
|
|
|
def test_namespace_first(self):
|
|
self.assertEqual(alerts.title_of({"alertname": "A", "namespace": "ns", "job": "j"}), "[alert] A (ns)")
|
|
|
|
def test_job_fallback(self):
|
|
self.assertEqual(alerts.title_of({"alertname": "TargetDown", "job": "pve"}), "[alert] TargetDown (pve)")
|
|
|
|
def test_cluster_fallback(self):
|
|
self.assertEqual(alerts.title_of({"alertname": "A"}), "[alert] A (cluster)")
|
|
|
|
|
|
class NoiseLog(unittest.TestCase):
|
|
def issues(self, *lines):
|
|
return [{"title": alerts.NOISE_TITLE, "body": "header\n\n" + "\n".join(lines)}]
|
|
|
|
def test_recent_entry_suppresses(self):
|
|
today = datetime.date.today().isoformat()
|
|
fps = alerts.noise_fingerprints(self.issues(f"- 0123456789abcdef KubeJobFailed — {today}: benign"))
|
|
self.assertEqual(fps, {"0123456789abcdef"})
|
|
|
|
def test_expired_entry_does_not_suppress(self):
|
|
old = (datetime.date.today() - datetime.timedelta(days=alerts.NOISE_TTL_DAYS + 1)).isoformat()
|
|
self.assertEqual(alerts.noise_fingerprints(self.issues(f"- 0123456789abcdef A — {old}: x")), set())
|
|
|
|
def test_invalid_date_is_dropped_not_fatal(self):
|
|
today = datetime.date.today().isoformat()
|
|
fps = alerts.noise_fingerprints(self.issues(
|
|
"- aaaaaaaaaaaaaaaa A — 2026-02-30: impossible date",
|
|
f"- bbbbbbbbbbbbbbbb B — {today}: fine",
|
|
))
|
|
self.assertEqual(fps, {"bbbbbbbbbbbbbbbb"})
|
|
|
|
def test_fingerprint_in_prose_does_not_suppress(self):
|
|
self.assertEqual(alerts.noise_fingerprints(self.issues("see cccccccccccccccc for details")), set())
|
|
|
|
def test_no_noise_issue(self):
|
|
self.assertEqual(alerts.noise_fingerprints([{"title": "other", "body": "x"}]), set())
|
|
|
|
|
|
class BlipFingerprint(unittest.TestCase):
|
|
def test_matches_old_pipeline(self):
|
|
# printf '%s|%s' "$NAME" "$SCOPE" | sha256sum | cut -c1-16
|
|
want = hashlib.sha256(b"TargetDown|pve").hexdigest()[:16]
|
|
series = [{"metric": {"alertname": "TargetDown", "job": "pve"}}]
|
|
with mock.patch.object(alerts, "http", side_effect=[[], {"data": {"result": series}}]), \
|
|
mock.patch.object(alerts, "open_issues", return_value=[]), \
|
|
mock.patch("builtins.print") as out:
|
|
alerts.cmd_gate(None)
|
|
printed = "\n".join(str(c.args[0]) for c in out.call_args_list)
|
|
self.assertIn(want, printed)
|
|
self.assertIn('"[alert] TargetDown (pve)"', printed)
|
|
|
|
|
|
class Gate(unittest.TestCase):
|
|
def run_gate(self, am, prom, issues):
|
|
with mock.patch.object(alerts, "http", side_effect=[am, prom]), \
|
|
mock.patch.object(alerts, "open_issues", return_value=issues), \
|
|
mock.patch("builtins.print") as out:
|
|
alerts.cmd_gate(None)
|
|
return [str(c.args[0]) for c in out.call_args_list]
|
|
|
|
def alert(self, name, ns, fp, receivers=("email",)):
|
|
return {"labels": {"alertname": name, "namespace": ns}, "fingerprint": fp,
|
|
"annotations": {}, "receivers": [{"name": r} for r in receivers]}
|
|
|
|
def test_nothing_new_sleeps(self):
|
|
am = [self.alert("Watchdog", "monitoring", "1" * 16),
|
|
self.alert("Muted", "x", "2" * 16, receivers=("null",)),
|
|
self.alert("Known", "ns", "3" * 16)]
|
|
lines = self.run_gate(am, {"data": {"result": []}}, [{"title": "[alert] Known (ns)", "body": ""}])
|
|
self.assertEqual(lines[-1], '{"wakeAgent": false}')
|
|
|
|
def test_new_alert_wakes(self):
|
|
lines = self.run_gate([self.alert("New", "ns", "4" * 16)], {"data": {"result": []}}, [])
|
|
self.assertNotIn('"wakeAgent": false', lines[-1])
|
|
self.assertIn('"[alert] New (ns)"', lines[-1])
|
|
|
|
def test_prometheus_error_keeps_live_alerts(self):
|
|
with mock.patch.object(alerts, "http",
|
|
side_effect=[[self.alert("New", "ns", "5" * 16)], alerts.ApiError("HTTP 503")]), \
|
|
mock.patch.object(alerts, "open_issues", return_value=[]), \
|
|
mock.patch("builtins.print") as out:
|
|
alerts.cmd_gate(None)
|
|
lines = [str(c.args[0]) for c in out.call_args_list]
|
|
self.assertTrue(any(line.startswith("WARNING: blip lookback unavailable") for line in lines))
|
|
self.assertIn('"[alert] New (ns)"', lines[-1])
|
|
|
|
|
|
class SecretGuard(unittest.TestCase):
|
|
def setUp(self):
|
|
self.dir = tempfile.TemporaryDirectory()
|
|
os.makedirs(os.path.join(self.dir.name, "gitea"))
|
|
with open(os.path.join(self.dir.name, "gitea", "api-token"), "w") as f:
|
|
f.write("0123456789abcdef0123456789abcdef01234567\n")
|
|
os.makedirs(os.path.join(self.dir.name, "talos.dev"))
|
|
with open(os.path.join(self.dir.name, "talos.dev", "config"), "w") as f:
|
|
f.write("context: x\nendpoints:\n - talos.default\nkey: LS0tLS1CRUdJTiBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0K\n")
|
|
self.patch = mock.patch.object(alerts, "SECRETS_DIR", self.dir.name)
|
|
self.patch.start()
|
|
|
|
def tearDown(self):
|
|
self.patch.stop()
|
|
self.dir.cleanup()
|
|
|
|
def test_whole_token_refused(self):
|
|
self.assertTrue(alerts.leaks_secret("token is 0123456789abcdef0123456789abcdef01234567 here"))
|
|
|
|
def test_piece_of_key_refused(self):
|
|
self.assertTrue(alerts.leaks_secret("leak: RUdJTiBFRDI1NTE5IFBSSVZB"))
|
|
|
|
def test_env_secret_refused(self):
|
|
with mock.patch.dict(os.environ, {"API_SERVER_KEY": "abcDEF123456ghiJKL789012mno"}):
|
|
self.assertTrue(alerts.leaks_secret("x abcDEF123456ghiJKL789012mno y"))
|
|
|
|
def test_unaligned_piece_refused(self):
|
|
# 24 characters starting at offset 3 of the 40-char token
|
|
self.assertTrue(alerts.leaks_secret("x 3456789abcdef0123456789a y"))
|
|
|
|
def test_certificates_are_not_secrets(self):
|
|
import base64
|
|
pem = "-----BEGIN CERTIFICATE-----\nMIIBszCCAVmgAwIBAgIRAKs1v0abcdefghijklmnop123456\n-----END CERTIFICATE-----\n"
|
|
with open(os.path.join(self.dir.name, "ca.crt"), "w") as f:
|
|
f.write(pem)
|
|
b64 = base64.b64encode(pem.encode()).decode()
|
|
with open(os.path.join(self.dir.name, "talos.dev", "config"), "a") as f:
|
|
f.write(f"ca: {b64}\n")
|
|
self.assertFalse(alerts.leaks_secret("caBundle: " + b64))
|
|
self.assertFalse(alerts.leaks_secret("MIIBszCCAVmgAwIBAgIRAKs1v0abcdefghijklmnop123456"))
|
|
# the private key in the same talosconfig is still guarded
|
|
self.assertTrue(alerts.leaks_secret("LS0tLS1CRUdJTiBFRDI1NTE5IFBSSVZBVEUgS0VZLS0tLS0K"))
|
|
|
|
def test_normal_body_passes(self):
|
|
body = ("**What is firing**: KubeJobFailed in renovate since 2026-10-20T10:00:00Z, "
|
|
"talos.default endpoint, kube-prometheus-stack-alertmanager.")
|
|
self.assertFalse(alerts.leaks_secret(body))
|
|
|
|
|
|
class FileWithLabel(unittest.TestCase):
|
|
BODY = "**What is firing**: KubeJobFailed in renovate since today, evidence below."
|
|
|
|
def run_file(self, labels_by_url, label="daedalus"):
|
|
calls = []
|
|
|
|
def fake(method, url, body=None, auth=False, timeout=30):
|
|
calls.append((method, url, body))
|
|
if method == "GET" and "/labels" in url:
|
|
return labels_by_url.get(url.split("?")[0], [])
|
|
if method == "GET":
|
|
return [] # open issues
|
|
return {"number": 9, "html_url": "https://gitea.renner.uno/konairius/homelab/issues/9"}
|
|
|
|
args = mock.Mock(title="[alert] KubeJobFailed (renovate)", fingerprint="0" * 16, label=label)
|
|
with mock.patch.object(alerts, "http", side_effect=fake), \
|
|
mock.patch.object(alerts, "leaks_secret", return_value=False), \
|
|
mock.patch("sys.stdin", mock.Mock(read=lambda: self.BODY)):
|
|
alerts.cmd_file(args)
|
|
return [c for c in calls if c[0] == "POST"][0][2]
|
|
|
|
def test_org_label_id_set_at_creation(self):
|
|
org = "https://gitea.renner.uno/api/v1/orgs/konairius/labels"
|
|
posted = self.run_file({org: [{"id": 41, "name": "atlas"}, {"id": 42, "name": "daedalus"}]})
|
|
self.assertEqual(posted["labels"], [42])
|
|
self.assertIn("`daedalus` expert", posted["body"])
|
|
self.assertNotIn("issue-fix", posted["body"])
|
|
|
|
def test_repo_label_wins(self):
|
|
repo = alerts.GITEA + "/labels"
|
|
posted = self.run_file({repo: [{"id": 7, "name": "Daedalus"}]})
|
|
self.assertEqual(posted["labels"], [7])
|
|
|
|
def test_missing_label_files_without_it(self):
|
|
posted = self.run_file({})
|
|
self.assertNotIn("labels", posted)
|
|
|
|
def test_label_is_required_and_validated(self):
|
|
with mock.patch("sys.argv", ["alerts.py", "file", "--title", "[alert] X (y)", "--fingerprint", "0" * 16,
|
|
"--label", "bogus"]), self.assertRaises(SystemExit):
|
|
alerts.main()
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|