test(monitoring): promtool suite for Uptime Kuma rules, terse comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(monitoring): watch off-site Uptime Kuma from Prometheus
2026-10-19 08:26:12 +02:00 · 2026-10-19 08:17:30 +02:00 · 2026-10-19 06:14:48 +00:00 · 2026-10-19 06:13:48 +00:00 · 2026-10-19 06:13:17 +00:00 · 2026-10-19 08:02:56 +02:00
23 changed files with 766 additions and 1 deletions
@@ -974,6 +974,71 @@ data:
          negate: false
          failure_result: false

+      # Hermes Agent dashboard (apps/hermes): its self-hosted OIDC login,
+      # also brokered for Hermes Desktop's native sign-in. Public client +
+      # PKCE (Hermes does not support confidential clients); RS256 so the
+      # dashboard can verify the ID token against the JWKS.
+      - id: hermes-users-group
+        model: authentik_core.group
+        identifiers:
+          name: Hermes Users
+        state: present
+        attrs:
+          is_superuser: false
+          # Members get the agent, which holds a read-only kubeconfig and
+          # the model login. Add/remove members HERE.
+          users:
+            - !Find [authentik_core.user, [username, k.renner]]
+
+      - id: hermes-provider
+        model: authentik_providers_oauth2.oauth2provider
+        identifiers:
+          name: Hermes Dashboard
+        state: present
+        attrs:
+          authentication_flow: null
+          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          client_type: public
+          client_id: hermes-dashboard
+          # See the Matrix (MAS) provider: a blueprint-made provider starts
+          # with no grant types.
+          grant_types:
+            - authorization_code
+            - refresh_token
+          signing_key: !Find [authentik_crypto.certificatekeypair, [name, authentik Self-signed Certificate]]
+          property_mappings:
+            - !Find [authentik_providers_oauth2.scopemapping, [managed, goauthentik.io/providers/oauth2/scope-openid]]
+            - !Find [authentik_providers_oauth2.scopemapping, [managed, goauthentik.io/providers/oauth2/scope-profile]]
+            - !Find [authentik_providers_oauth2.scopemapping, [managed, goauthentik.io/providers/oauth2/scope-email]]
+          redirect_uris:
+            - url: https://hermes.renner.uno/auth/callback
+              matching_mode: strict
+
+      # The slug is the issuer path: https://auth.renner.uno/application/o/hermes/
+      - model: authentik_core.application
+        identifiers:
+          slug: hermes
+        state: present
+        attrs:
+          name: Hermes
+          slug: hermes
+          provider: !KeyOf hermes-provider
+          meta_launch_url: https://hermes.renner.uno
+          meta_description: "Hermes Agent dashboard"
+
+      # Only the Hermes Users group passes.
+      - model: authentik_policies.policybinding
+        identifiers:
+          target: !Find [authentik_core.application, [slug, hermes]]
+          group: !KeyOf hermes-users-group
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+          failure_result: false
+
      # OpenProject, decommissioned. Application first: Application.provider
      # is on_delete=PROTECT. Its policy binding cascades with it.
      - model: authentik_core.application
@@ -0,0 +1,32 @@
+# Bearer key for the OpenAI-compatible API server (hermes-api.renner.uno).
+# Generated once in-cluster, nothing in OpenBao. Read it with
+#   kubectl -n hermes get secret hermes-api-key -o jsonpath='{.data.API_SERVER_KEY}' | base64 -d
+apiVersion: generators.external-secrets.io/v1alpha1
+kind: Password
+metadata:
+  name: hermes-api-key
+  namespace: hermes
+spec:
+  length: 64
+  digits: 16
+  symbols: 0
+  allowRepeat: true
+  noUpper: false
+  secretKeys:
+    - API_SERVER_KEY
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hermes-api-key
+  namespace: hermes
+spec:
+  refreshPolicy: CreatedOnce
+  target:
+    name: hermes-api-key
+  dataFrom:
+    - sourceRef:
+        generatorRef:
+          apiVersion: generators.external-secrets.io/v1alpha1
+          kind: Password
+          name: hermes-api-key
@@ -0,0 +1,170 @@
+# All Hermes state: config.yaml, .env (keys entered in the dashboard), auth.json
+# (the Anthropic OAuth login), sessions, memories, skills, cron. Mounted at
+# /opt/data, the image's HERMES_HOME.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: hermes-data
+  namespace: hermes
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 10Gi
+---
+# Hermes Agent: the gateway (messaging platforms, cron, API server on 8642) and
+# the dashboard (9119, also the Hermes Desktop remote-gateway backend), both
+# supervised by the image's s6-overlay. s6 must be PID 1 and starts as root,
+# remapping and dropping to the `hermes` user (10000).
+#
+# One replica, Recreate: two gateways must never share one data directory.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: hermes
+  namespace: hermes
+  labels:
+    app.kubernetes.io/name: hermes
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: hermes
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: hermes
+    spec:
+      serviceAccountName: hermes
+      securityContext:
+        fsGroup: 10000
+        seccompProfile:
+          type: RuntimeDefault
+      initContainers:
+        # The dashboard trusts X-Forwarded-* only from listed peers; Traefik
+        # connects from the pod network. No env equivalent exists, so it is
+        # set in config.yaml on every start (idempotent).
+        - name: config
+          image: nousresearch/hermes-agent:v2026.9.24 # {"$imagepolicy": "flux-system:hermes"}
+          command:
+            - /opt/hermes/.venv/bin/hermes
+            - config
+            - set
+            - dashboard.trusted_proxies
+            - '["10.244.0.0/16","fd10:244::/48"]'
+          env:
+            - name: HERMES_HOME
+              value: /opt/data
+            - name: HOME
+              value: /opt/data
+          securityContext:
+            runAsUser: 10000
+            runAsGroup: 10000
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          volumeMounts:
+            - name: data
+              mountPath: /opt/data
+        # kubectl for the agent. /opt/data/.local/bin is on the image's PATH;
+        # an emptyDir there keeps the binary out of the PVC and current with
+        # this pin.
+        - name: kubectl
+          image: docker.io/alpine/kubectl:1.37.1@sha256:b568e8beb6a79aee269c142343078a278b85e7710f1c0e7d8d6b21d1eb566a48
+          command: ["cp", "/usr/local/bin/kubectl", "/tools/kubectl"]
+          securityContext:
+            runAsUser: 10000
+            runAsGroup: 10000
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          volumeMounts:
+            - name: tools
+              mountPath: /tools
+      containers:
+        - name: hermes
+          image: nousresearch/hermes-agent:v2026.9.24 # {"$imagepolicy": "flux-system:hermes"}
+          args: ["gateway", "run"]
+          env:
+            - name: HERMES_DASHBOARD
+              value: "1"
+            - name: HERMES_DASHBOARD_PUBLIC_URL
+              value: https://hermes.renner.uno
+            # Dashboard login: Authentik OIDC, public PKCE client (blueprint
+            # in apps/authentik/blueprints-configmap.yaml, group Hermes Users).
+            - name: HERMES_DASHBOARD_OIDC_ISSUER
+              value: https://auth.renner.uno/application/o/hermes/
+            - name: HERMES_DASHBOARD_OIDC_CLIENT_ID
+              value: hermes-dashboard
+            - name: API_SERVER_ENABLED
+              value: "true"
+            - name: API_SERVER_HOST
+              value: 0.0.0.0
+          envFrom:
+            - secretRef:
+                name: hermes-api-key
+          ports:
+            - name: dashboard
+              containerPort: 9119
+            - name: api
+              containerPort: 8642
+          # The dashboard is the user-facing surface; /api/status is public.
+          startupProbe:
+            httpGet:
+              path: /api/status
+              port: dashboard
+            periodSeconds: 10
+            failureThreshold: 30
+          readinessProbe:
+            httpGet:
+              path: /api/status
+              port: dashboard
+            periodSeconds: 15
+          livenessProbe:
+            httpGet:
+              path: /api/status
+              port: dashboard
+            periodSeconds: 60
+            timeoutSeconds: 10
+            failureThreshold: 3
+          resources:
+            requests:
+              cpu: 100m
+              memory: 512Mi
+            limits:
+              memory: 4Gi
+          volumeMounts:
+            - name: data
+              mountPath: /opt/data
+            - name: tools
+              mountPath: /opt/data/.local/bin
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: hermes-data
+        - name: tools
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: hermes
+  namespace: hermes
+  labels:
+    app.kubernetes.io/name: hermes
+spec:
+  selector:
+    app.kubernetes.io/name: hermes
+  ports:
+    - name: dashboard
+      port: 9119
+      targetPort: dashboard
+    - name: api
+      port: 8642
+      targetPort: api
@@ -0,0 +1,59 @@
+# Both hosts are LAN-only (no dns.renner.uno/public or netbird.renner.uno/public
+# label).
+#
+# hermes.renner.uno — dashboard and Hermes Desktop remote gateway. Plain
+# Traefik, not the authentik-gate class: Hermes Desktop calls the API with a
+# Bearer token from its native (PKCE) sign-in, which a forward-auth gate would
+# redirect. Authentik still guards it: the dashboard's own OIDC login against
+# the `hermes` application, restricted to the Hermes Users group.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: hermes
+  namespace: hermes
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - hermes.renner.uno
+      secretName: hermes-tls
+  rules:
+    - host: hermes.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: hermes
+                port:
+                  number: 9119
+---
+# hermes-api.renner.uno — the OpenAI-compatible API server. Hermes checks
+# `Authorization: Bearer <API_SERVER_KEY>` itself (secret hermes-api-key).
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: hermes-api
+  namespace: hermes
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - hermes-api.renner.uno
+      secretName: hermes-api-tls
+  rules:
+    - host: hermes-api.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: hermes
+                port:
+                  number: 8642
@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - rbac.yaml
+  - api-key.yaml
+  - hermes.yaml
+  - ingress.yaml
@@ -0,0 +1,22 @@
+# Read-only cluster access for the agent's kubectl, as for claude-workstation:
+# the built-in `view` ClusterRole, which excludes Secrets. The token is the
+# pod's projected ServiceAccount token; kubectl picks it up as in-cluster
+# config.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: hermes
+  namespace: hermes
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: hermes-view
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: view
+subjects:
+  - kind: ServiceAccount
+    name: hermes
+    namespace: hermes
@@ -18,6 +18,7 @@ resources:
  - prometheusrule-netbird-availability.yaml
  - prometheusrule-netbird-ingress.yaml
  - prometheusrule-openbao.yaml
+  - prometheusrule-uptime-kuma.yaml
  # Prometheus's own retention. Until REN-107 `retention: 90d` and
  # `retentionSize: 30GB` were both set, the size cap bound first, and the
  # effective window was a 20.5-27.2 day saw-tooth with nothing reporting it.
@@ -44,6 +44,8 @@ spec:
        # traverses the WAN and detects a controller/DC outage (the
        # 2026-07-24 one ran ~20 h with no signal). Dashboard root, 200.
        - https://netbird.renner.uno
+        # Dead-man probe for the off-site Uptime Kuma.
+        - https://kuma.renner.uno
        # NetBird relay health endpoints on the netcup members (relay
        # binary serves :9000/health, verified on 0.73.2). Plain http,
        # ports outside the hairpin's 80/443 — real WAN path.
@@ -26,6 +26,7 @@ spec:
        #   proxy-1/2:9000/health         same file, relay redundancy
        #   ci.renner.uno                 WoodpeckerDown
        #   openbao .../sys/health        OpenBaoUnavailable
+        #   kuma.renner.uno               UptimeKumaUnreachable
        #
        # Warning, not critical, on purpose: this rule cannot know what a given
        # target means. A target whose loss deserves a critical page gets its
@@ -33,7 +34,7 @@ spec:
        # exclusion list from being arbitrary.
        # anchor: blackbox-probe-failing
        - alert: BlackboxProbeFailing
-          expr: probe_success{job="blackbox-http", instance!~"https://fsn1\\.your-objectstorage\\.com|https://netbird\\.renner\\.uno|http://proxy-[12]\\.renner\\.uno:9000/health|https://ci\\.renner\\.uno|https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
+          expr: probe_success{job="blackbox-http", instance!~"https://fsn1\\.your-objectstorage\\.com|https://netbird\\.renner\\.uno|http://proxy-[12]\\.renner\\.uno:9000/health|https://ci\\.renner\\.uno|https://kuma\\.renner\\.uno|https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
          for: 10m
          annotations:
            summary: "Blackbox probe of {{ $labels.instance }} has been failing for 10 minutes"
@@ -0,0 +1,35 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: uptime-kuma
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus selects on this label.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: uptime-kuma.rules
+      rules:
+        # Dead-man probe for the off-site Uptime Kuma.
+        - alert: UptimeKumaUnreachable
+          expr: probe_success{job="blackbox-http", instance="https://kuma.renner.uno"} == 0
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Uptime Kuma (kuma.renner.uno) unreachable for 10m+"
+            description: "The blackbox probe of the off-site Uptime Kuma has been failing for 10 minutes. The external endpoints it monitors are unwatched until it returns. Check the NetBird VPS (netbird.renner.uno), the uptime-kuma container and Traefik."
+
+        # monitor_response_time is ms; -1 when a beat has no ping (DOWN, group).
+        # anchor: uptime-kuma-slow-response
+        - alert: UptimeKumaSlowResponse
+          expr: avg by (monitor_name) (avg_over_time((monitor_response_time > 0)[15m:1m])) > 1500
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Uptime Kuma monitor {{ $labels.monitor_name }} averaging {{ $value | humanize }} ms response time"
+            description: >-
+              The average response time of monitor {{ $labels.monitor_name }},
+              as measured from the off-site Uptime Kuma, has stayed above
+              1500 ms for 15 minutes.
@@ -32,6 +32,7 @@ retired the depth rule in favour of one persistence rule (`> 5` for `20m`).
 | `ceph_test.yaml` | Hand-written suite for the `CephPoolLostRedundancyBlind` canary in `prometheusrule-ceph.yaml` (REN-93). |
 | `disk-wear_test.yaml` | Hand-written suite for the EPHEMERAL exclusion in `prometheusrule-disk-wear.yaml` (REN-419). |
 | `gitea_test.yaml` | Hand-written suite for `GiteaDataVolumeLow` in `prometheusrule-gitea.yaml` (REN-583). |
+| `uptime-kuma_test.yaml` | Hand-written suite for `prometheusrule-uptime-kuma.yaml` and the kuma exclusion in `prometheusrule-blackbox.yaml`. |
 | `mutations.txt` | Negative controls. Each must make the suite fail. |

 ## Adding a suite
@@ -172,3 +172,12 @@ disk-wear-ephemeral-keeps-partition|kubernetes/apps/monitoring-rules/prometheusr
 # exactly-25% case in gitea_test.yaml.
 gitea-for-10m|kubernetes/apps/monitoring-rules/prometheusrule-gitea.yaml|/alert: GiteaDataVolumeLow/,/for:/s/for: 15m/for: 10m/
 gitea-floor-le|kubernetes/apps/monitoring-rules/prometheusrule-gitea.yaml|s/^\([[:space:]]*\)< 0\.25$/\1<= 0.25/
+
+# prometheusrule-uptime-kuma.yaml / blackbox kuma exclusion. Detected by the
+# cases in uptime-kuma_test.yaml.
+kuma-slow-no-positive-filter|kubernetes/apps/monitoring-rules/prometheusrule-uptime-kuma.yaml|s/(monitor_response_time > 0)/(monitor_response_time)/
+kuma-slow-threshold-2500|kubernetes/apps/monitoring-rules/prometheusrule-uptime-kuma.yaml|s/\[15m:1m\])) > 1500$/[15m:1m])) > 2500/
+kuma-slow-threshold-900|kubernetes/apps/monitoring-rules/prometheusrule-uptime-kuma.yaml|s/\[15m:1m\])) > 1500$/[15m:1m])) > 900/
+kuma-slow-for-30m|kubernetes/apps/monitoring-rules/prometheusrule-uptime-kuma.yaml|/alert: UptimeKumaSlowResponse/,/for:/s/for: 15m/for: 30m/
+kuma-down-for-20m|kubernetes/apps/monitoring-rules/prometheusrule-uptime-kuma.yaml|/alert: UptimeKumaUnreachable/,/for:/s/for: 10m/for: 20m/
+kuma-blackbox-exclusion-dropped|kubernetes/apps/monitoring-rules/prometheusrule-blackbox.yaml|s/kuma\\\\\.renner/kumx\\\\.renner/
@@ -0,0 +1,110 @@
+# Suite for prometheusrule-uptime-kuma.yaml and the kuma exclusion in
+# prometheusrule-blackbox.yaml.
+rule_files:
+  - ../prometheusrule-uptime-kuma.yaml
+  - ../prometheusrule-blackbox.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # 2000 ms sustained: pending at 0m, fires 15m later, not before.
+  - interval: 1m
+    input_series:
+      - series: 'monitor_response_time{monitor_name="slow-mon"}'
+        values: '2000+0x40'
+    alert_rule_test:
+      - eval_time: 14m
+        alertname: UptimeKumaSlowResponse
+        exp_alerts: []
+      - eval_time: 16m
+        alertname: UptimeKumaSlowResponse
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              monitor_name: slow-mon
+            exp_annotations:
+              summary: "Uptime Kuma monitor slow-mon averaging 2k ms response time"
+              description: "The average response time of monitor slow-mon, as measured from the off-site Uptime Kuma, has stayed above 1500 ms for 15 minutes."
+
+  # 1000 ms stays silent; a monitor reporting only -1 (DOWN, group) stays silent.
+  - interval: 1m
+    input_series:
+      - series: 'monitor_response_time{monitor_name="fast-mon"}'
+        values: '1000+0x40'
+      - series: 'monitor_response_time{monitor_name="down-mon"}'
+        values: '-1+0x40'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: UptimeKumaSlowResponse
+        exp_alerts: []
+
+  # Alternating 1000 and -1: the -1 samples must not drag a slow average down
+  # nor count as slow; positives average 1000, silent.
+  - interval: 1m
+    input_series:
+      - series: 'monitor_response_time{monitor_name="flap-fast"}'
+        values: '1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1 1000 -1'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: UptimeKumaSlowResponse
+        exp_alerts: []
+
+  # Alternating 2000 and -1: the -1 samples are excluded, so the average of the
+  # real samples is 2000 and the alert fires. Unfiltered, the mean is ~500.
+  - interval: 1m
+    input_series:
+      - series: 'monitor_response_time{monitor_name="flap-slow"}'
+        values: '2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1 2000 -1'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: UptimeKumaSlowResponse
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              monitor_name: flap-slow
+            exp_annotations:
+              summary: "Uptime Kuma monitor flap-slow averaging 2k ms response time"
+              description: "The average response time of monitor flap-slow, as measured from the off-site Uptime Kuma, has stayed above 1500 ms for 15 minutes."
+
+  # Dead-man probe: critical after 10m of failure; the catch-all warning must
+  # not also fire for kuma, but still fires for another failing instance.
+  - interval: 1m
+    input_series:
+      - series: 'probe_success{job="blackbox-http",instance="https://kuma.renner.uno"}'
+        values: '0+0x40'
+      - series: 'probe_success{job="blackbox-http",instance="https://other.renner.uno"}'
+        values: '0+0x40'
+    alert_rule_test:
+      - eval_time: 9m
+        alertname: UptimeKumaUnreachable
+        exp_alerts: []
+      - eval_time: 11m
+        alertname: UptimeKumaUnreachable
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              job: blackbox-http
+              instance: https://kuma.renner.uno
+            exp_annotations:
+              summary: "Uptime Kuma (kuma.renner.uno) unreachable for 10m+"
+              description: "The blackbox probe of the off-site Uptime Kuma has been failing for 10 minutes. The external endpoints it monitors are unwatched until it returns. Check the NetBird VPS (netbird.renner.uno), the uptime-kuma container and Traefik."
+      - eval_time: 15m
+        alertname: BlackboxProbeFailing
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: blackbox-http
+              instance: https://other.renner.uno
+            exp_annotations:
+              summary: "Blackbox probe of https://other.renner.uno has been failing for 10 minutes"
+              description: "The endpoint is not answering as expected. For the Authentik-gated hostnames this is a reachability check covering DNS, TLS, Traefik and the gate outpost — it says nothing about the app behind the gate, which the internal /-/healthy probes cover separately."
+
+  # Healthy probe stays silent.
+  - interval: 1m
+    input_series:
+      - series: 'probe_success{job="blackbox-http",instance="https://kuma.renner.uno"}'
+        values: '1+0x40'
+    alert_rule_test:
+      - eval_time: 30m
+        alertname: UptimeKumaUnreachable
+        exp_alerts: []
@@ -97,6 +97,10 @@ spec:
              resources:
                requests:
                  storage: 50Gi
+        # Mounted at /etc/prometheus/secrets/<name>/; see
+        # ./uptime-kuma-metrics-externalsecret.yaml.
+        secrets:
+          - uptime-kuma-metrics
        # Additional scrape targets not covered by ServiceMonitors.
        additionalScrapeConfigs:
          # Gitea exposes /metrics anonymously on its main HTTP port. The
@@ -106,6 +110,16 @@ spec:
            static_configs:
              - targets: ["gitea.gitea.svc.cluster.local:3000"]
            metrics_path: /metrics
+          # Off-site Uptime Kuma. /metrics takes the API key as the basic-auth
+          # password with an empty username.
+          - job_name: uptime-kuma
+            scheme: https
+            metrics_path: /metrics
+            basic_auth:
+              username: ""
+              password_file: /etc/prometheus/secrets/uptime-kuma-metrics/api_key
+            static_configs:
+              - targets: ["kuma.renner.uno:443"]

    # Grafana
    grafana:
@@ -3,6 +3,7 @@ kind: Kustomization
 resources:
  - grafana-oauth-externalsecret.yaml
  - alertmanager-smtp-externalsecret.yaml
+  - uptime-kuma-metrics-externalsecret.yaml
  - helmrelease-prometheus.yaml
  - helmrelease-loki.yaml
  - helmrelease-alloy-syslog.yaml
@@ -0,0 +1,19 @@
+---
+# Uptime Kuma /metrics API key for Prometheus; OpenBao homelab/infrastructure/uptime-kuma-metrics.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: uptime-kuma-metrics
+  namespace: monitoring
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: uptime-kuma-metrics
+  data:
+    - secretKey: api_key
+      remoteRef:
+        key: infrastructure/uptime-kuma-metrics
+        property: api_key
@@ -0,0 +1,18 @@
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: hermes
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/hermes
+  prune: true
+  wait: true
+  dependsOn:
+    - name: infrastructure
+    - name: infrastructure-crds
@@ -0,0 +1,24 @@
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImageRepository
+metadata:
+  name: hermes
+  namespace: flux-system
+spec:
+  image: nousresearch/hermes-agent
+  interval: 1h
+---
+# v-prefix tags (v0.22.2). The filter skips rc.*, -desktop and per-arch tags.
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImagePolicy
+metadata:
+  name: hermes
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: hermes
+  filterTags:
+    pattern: '^v(?P<ver>\d+\.\d+\.\d+)$'
+    extract: '$ver'
+  policy:
+    semver:
+      range: ">=0.22.2"
@@ -10,6 +10,7 @@ resources:
  - gatus.yaml
  - gitea-mcp.yaml
  - gotenberg.yaml
+  - hermes.yaml
  - home-assistant.yaml
  - immich.yaml
  - kuma-sync.yaml
@@ -294,3 +294,12 @@ apiVersion: v1
 kind: Namespace
 metadata:
  name: kuma-sync
+---
+# Hermes Agent (kubernetes/apps/hermes). Declared here so it exists before
+# infrastructure-crds applies its CNPs. No pod-security labels: the image's
+# s6-overlay init starts as root and drops to the `hermes` user, which the
+# cluster's default `baseline` enforcement allows.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: hermes
@@ -0,0 +1,137 @@
+---
+# Baseline policy for the `hermes` namespace (Hermes Agent, apps/hermes).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny
+  namespace: hermes
+spec:
+  endpointSelector: {}
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": hermes
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    - fromEntities:
+        - host
+        - remote-node
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": hermes
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+            - port: "53"
+              protocol: TCP
+    # Every LAN hostname (gitea, home, auth for the dashboard's OIDC
+    # discovery, matrix) via Traefik.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+---
+# The agent itself: model APIs (api.anthropic.com), web tools, messaging
+# platforms, read-only kubectl, and the monitoring read APIs. The matching
+# ingress grant on the monitoring side is monitoring-ns.yaml
+# (monitoring-allow-hermes).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: hermes-egress
+  namespace: hermes
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: hermes
+  egress:
+    - toEntities:
+        - world
+      toPorts:
+        - ports:
+            - port: "443"
+              protocol: TCP
+            - port: "80"
+              protocol: TCP
+    # `kube-apiserver` is its own entity in Cilium; port is the post-DNAT 6443.
+    - toEntities:
+        - kube-apiserver
+      toPorts:
+        - ports:
+            - port: "6443"
+              protocol: TCP
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+            app.kubernetes.io/name: prometheus
+      toPorts:
+        - ports:
+            - port: "9090"
+              protocol: TCP
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+            app.kubernetes.io/name: alertmanager
+      toPorts:
+        - ports:
+            - port: "9093"
+              protocol: TCP
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+            app.kubernetes.io/name: loki
+      toPorts:
+        - ports:
+            - port: "3100"
+              protocol: TCP
+---
+# Traefik serves the dashboard (9119) and the OpenAI-compatible API (8642).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: hermes-allow-traefik
+  namespace: hermes
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: hermes
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "9119"
+              protocol: TCP
+            - port: "8642"
+              protocol: TCP
@@ -34,5 +34,6 @@ resources:
  - uptime-monitor-ns.yaml
  - kuma-sync-ns.yaml
  - renovate-ns.yaml
+  - hermes-ns.yaml
  - allow-cilium-health.yaml
  - kube-system-hubble-relay.yaml
@@ -57,3 +57,30 @@ spec:
            - { type: 135, family: IPv6 }
            - { type: 136, family: IPv6 }
            - { type: 143, family: IPv6 }
+---
+# Hermes Agent (hermes ns) reads metrics, alerts and logs: the Prometheus,
+# Alertmanager and Loki HTTP APIs, scoped to those pods and ports.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: monitoring-allow-hermes
+  namespace: monitoring
+spec:
+  endpointSelector:
+    matchExpressions:
+      - key: app.kubernetes.io/name
+        operator: In
+        values: [prometheus, alertmanager, loki]
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": hermes
+            app.kubernetes.io/name: hermes
+      toPorts:
+        - ports:
+            - port: "9090"
+              protocol: TCP
+            - port: "9093"
+              protocol: TCP
+            - port: "3100"
+              protocol: TCP
Author	SHA1	Message	Date
claude-botandClaude Opus 5.5	72927293e2	test(monitoring): promtool suite for Uptime Kuma rules, terse comments ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details review-bot skipped: draft Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-19 08:26:12 +02:00
claude-botandClaude Opus 5.5	a35bc6b91d	feat(monitoring): watch off-site Uptime Kuma from Prometheus Add a blackbox-http probe of kuma.renner.uno with a dedicated critical UptimeKumaUnreachable rule (excluded from BlackboxProbeFailing), scrape Kuma /metrics with the API key from OpenBao, and alert on sustained slow monitor response times. No egress policy exists in the monitoring namespace, so no CNP change is needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-19 08:17:30 +02:00
k.renner	e614c748b9	Merge pull request 'fix(kuma-sync): create when the previous monitor ID belongs to another owner' (#1354 ) from fix/kuma-sync-foreign-key-create into master ci/woodpecker/cron/alert-monitor Pipeline was successful Details ci/woodpecker/push/build-ci-images Pipeline was successful Details Reviewed-on: k.renner/homelab#1354	2026-10-19 06:14:48 +00:00
security-bot	30b0f43911	chore(image-automation): bump nousresearch/hermes-agent:v2026.9.24 Updates managed by image-automation-controller.	2026-10-19 06:13:48 +00:00
k.renner	10819fac62	Merge pull request 'feat(hermes): run Hermes Agent in the cluster' (#1352 ) from feat/hermes-agent into master Reviewed-on: k.renner/homelab#1352	2026-10-19 06:13:17 +00:00
claude-bot	dd954ce23a	feat(hermes): run Hermes Agent in the cluster ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details review-bot reviewed: request changes Details Hermes Agent as a pod in its own `hermes` namespace: gateway (messaging platforms, cron, OpenAI-compatible API) and dashboard, supervised by the image's s6-overlay, state on a 10Gi ceph-block PVC. - hermes.renner.uno (LAN): dashboard and Hermes Desktop remote gateway. Plain Traefik plus the dashboard's own Authentik OIDC login (public PKCE client, Hermes Users group) — a forward-auth gate would break Desktop's Bearer-token calls. - hermes-api.renner.uno (LAN): API server, Bearer key generated in-cluster by an ESO Password generator. - Read-only kubectl via the built-in `view` ClusterRole. - CNPs: world 443/80, kube-apiserver, Traefik for LAN hostnames, and Prometheus/Alertmanager/Loki (with the matching monitoring-side grant). - Image automation for nousresearch/hermes-agent v-tags. After deploy: log in to Anthropic inside the pod (`hermes auth add anthropic --type oauth --no-browser`, then `hermes model`); platform and Home Assistant tokens go in via the dashboard. Not yet covered: a Velero schedule for the PVC.	2026-10-19 08:02:56 +02:00