feat(monitoring): scrape the ESO controller metrics (REN-103, merge A) #996
Closed
Ghost
wants to merge 1 commits from
atlas/ren-103-eso-metrics-scrape into main
pull from: atlas/ren-103-eso-metrics-scrape
merge into: :main
:main
:fix/gate-embedded-refresh-interval
:fix/gate-embedded-browser-host
:feat/authentik-gate-forwardauth
:docs/drop-wildcard-references
:fix/netbird-controller-disable-dns
:feat/dns-labels-split-2
:feat/dns-labels-split-1
:fix/cloudflare-lb-sync-image-tag
:feat/cloudflare-lb-sync
:fix/wordpress-php85-php2shell
:feat/update-os
:chore/talos-1.13.12
:atlas/talos-v1.13.12-cve-2026-99999
:fix/gitea-webhook-sync-org-scope
:atlas/kuma-mcb-www-retries
:feat/hermes-gitea-pr-review-loop
:atlas/traefik-chart-42
:fix/monitoring-k8s-gateway-probe
:alert-fix/issue-1081-rotator
:ci-image/claude-2.1.306
:alert-fix/issue-1084-v2
:atlas/flux-2.9.6
:alert-fix/issue-1082
:feat/hermes-gitea-issues-kanban
:worktree-restore-test-real
:fix/hermes-multiplex-env
:feat/hermes-experts
:feat/hermes-ha-mcp
:feat/kuma-status-page
:fix/hermes-ha-updates-review
:feat/hermes-alert-monitor
:feat/hermes-ha-updates
:fix/turn-tcp-router
:fix/crowdsec-own-ca
:fix/livekit-service-links
:fix/crowdsec-lapi-transport-ca
:feat/matrix-element-call
:feat/kuma-public-coverage
:feat/ha-matrix-nabu-secret
:feat/hermes-matrix-cross-signing
:feat/hermes-matrix
:fix/wiki-basic-auth-in-service
:proposal/traefik-vip-maglev-dsr
:fix/hermes-dashboard-ws-ping-timeout
:fix/hermes-dashboard-ws-ping-interval
:fix/hermes-dashboard-refresh-token
:feat/wiki-drop-gate
:feat/wiki-oidc-provider
:fix/hermes-soul-talos-node-names
:claude/docs-konairius-main
:claude/drop-k-renner-org-secrets
:claude/prompts-main
:claude/default-branch-main
:feat/hermes-cli-logins
:claude/plan-step0-claude-bot
:alert-fix/issue-1363
:claude/homelab-konairius
:claude/konairius-org-secrets
:fix/hermes-image-registry-bot
:feat/hermes-use-custom-image
:feat/hermes-image
:feat/kuma-task5-monitors
:feat/hermes-gitea
:fix/hermes-configmap-namespace
:feat/hermes-homelab-context
:fix/uptime-kuma-script-mount
:feat/kuma-task4-bootstrap
:feat/kuma-task4-cluster
:feat/kuma-sync-default-notifications
:fix/hermes-image-policy
:fix/kuma-sync-foreign-key-create
:feat/hermes-agent
:chore/kuma-sync-smoke-remove
:feat/kuma-sync-smoke
:feat/kuma-sync-deploy
:ci-image/claude-2.1.303
:renovate/authentik
:renovate/home-assistant-minor-patch
:renovate/pin-image-digests
:feat/kuma-sync
:docs/uptime-kuma-plan
:chore/remove-autokuma
:fix/uptime-kuma-ipv6
:feat/autokuma-pilot-update
:fix/drop-flux-image-automation-alert
:feat/autokuma-pilot
:feat/autokuma
:fix/uptime-kuma-fail-guard
:fix/uptime-kuma-setup-diagnostics
:feat/uptime-kuma-vps
:ci/ansible-image-hvac
:claude/plan-homelab-konairius-main
:fix/kubejobfailed-resolve-on-later-success
:claude/homeassistant-konairius
:claude/ha-config-sync-identity
:feat/site-base-domain
:claude/review-bot-repos-konairius
:claude/authentik-konairius-images
:claude/notes-renovate-konairius
:claude/vsftpd-konairius-image
:claude/registry-bot-read-scope
:claude/registry-bot
:ci-image/claude-2.1.302
:claude/review-bot-approved
:claude/review-bot-backends
:fix/wp-pg-recovery-source
:feat/review-bot-status
:worktree-hcloud-wg-site
:node-dns-via-k8s-gateway
:chore/remove-paperclip
:worktree-k8s-gateway-chart-4
:fix/review-bot-approve-with-unverified
:chore/remove-wiki-maintenance-ci
:fix/review-bot-unique-tag
:fix/review-bot-rereview
:fix/review-bot-build-trigger
:fix/review-bot-backoff-cap
:fix/review-bot-rate-limit
:chore/disable-old-pr-review
:feat/review-bot-deploy
:feat/review-bot-service
:feat/pr-review-mistral
:fix/gitea-mcp-public-gitea-url
:alert-fix/issue-1242
:alert-fix/issue-1241
:fix/gitea-mcp-command
:feat/gitea-mcp
:claude/netbird-service-sync-pin
:claude/gitea-public
:feat/embed-knn-bench
:claude/pr-review-latest-cli
:ci-image/claude-2.1.296
:chore/decommission-openproject
:claude/pr-review-central
:feat/openproject-authentik-oidc
:fix/alloy-syslog-udm-cef-regex
:alert-fix/issue-1217
:alert-fix/issue-1214
:atlas/ren-1363-openproject-seeder-order
:atlas/ren-1367-gitea-1.27
:atlas/ren-1367-gitea-pre-upgrade-snapshot
:atlas/ren-1363-openproject
:atlas/ren-1345-ops-image-jq
:atlas/ren-1345-hermes-prometheus-read
:atlas/ren-1345-bridge-agent-secrets
:atlas/ren-802-retire-sml2mqtt
:atlas/ren-953-remove-legacy-users
:atlas/ren-1307-edge1-register
:atlas/ren-1307-edge1-cred-receiver
:atlas/ren-953-controller-ha-cert
:atlas/ren-1256-prune-shellies-secret
:atlas/ren-953-ha-mqtt-cert-sync
:atlas/ren-1256-adopt-shellies-secret
:atlas/ren-1203-eso-gitea-namespace
:atlas/ren-1203-gitea-package-retention
:atlas/ren-1196-agent-runtime-tzdata
:atlas/ren-1017-remove-legacy-shellies
:atlas/ren-1017-ht-wake-reland
:atlas/ren-1017-shelly-30
:atlas/ren-1167-revert-1183
:alert-fix/issue-1184
:atlas/ren-1017-ht-wake
:atlas/ren-1017-shelly-group5c
:atlas/ren-1017-shelly-group5b
:atlas/ren-1080-controller-1d03144-rotate-hold
:atlas/ren-1017-shelly-group5
:atlas/ren-1017-shelly-group4
:atlas/ren-1053-controller-a307693
:atlas/ren-1017-shelly-group3
:atlas/ren-1017-shelly-group2
:atlas/ren-1017-shelly-group1
:atlas/ren-1017-device-list
:atlas/ren-1017-shelly-accounts
:atlas/ren-953-agents-read-paperclip-pod
:atlas/ren-953-agents-read
:atlas/ren-953-ha-mqtt-tls
:atlas/ren-953-ha-account
:atlas/ren-953-evcc-cert
:atlas/ren-953-evcc-account
:atlas/ren-953-z2m-cert
:atlas/ren-953-z2m-account
:atlas/ren-953-health-probe-cert
:atlas/ren-931-agent-shelly-manager
:atlas/ren-900-controller-bump
:atlas/ren-900-mosquitto-dynsec
:atlas/ren-888-revoke-test-account
:atlas/ren-888-mqtt-account-controller
:atlas/ren-841-remove-repair-job
:atlas/ren-841-k8s-plugin-termination-receipt
:atlas/ren-841-lease-receipt-repair
:atlas/ren-802-suspend-open3e-int-rotator
:atlas/ren-826-remove-read-job
:atlas/ren-826-room-setpoint-read
:atlas/ren-809-remove-capture
:atlas/ren-809-open3e-fbh-capture
:atlas/ren-723-authentik-2026.8.3
:atlas/ren-772-remove-rekey-job
:atlas/ren-772-wp-repo4-rekey
:atlas/ren-722-mdns-relay-v191-attach
:atlas/ren-723-openbao-rolling
:atlas/ren-722-dhcp-lease-renew
:atlas/ren-722-cni-plugins-installer-v1.9.1
:atlas/ren-723-influxdb-snapshot
:atlas/ren-722-cni-plugins-v1.9.1
:atlas/ren-722-job-force-annotations
:atlas/ren-717-matrix-bridge-1791195981
:atlas/ren-702-paperclip-cap-kill
:atlas/ren-691-matrix-bridge-1791179547
:atlas/ren-685-matrix-bridge-1791177680
:ci-image/claude-2.1.289
:atlas/ren-671-matrix-bridge-1791152484
:atlas/ren-663-matrix-public
:atlas/ren-587-matrix-bridge
:atlas/ren-567-authentik-grant-types
:atlas/ren-567-mas
:atlas/ren-627-edge1-ssh-egress
:atlas/ren-621-loki-read-gateway
:atlas/ren-603-pclp-pg-memory
:atlas/ren-583-gitea-pvc-volumename
:atlas/ren-583-gitea-volume-60gi
:atlas/ren-563-claude-session-resume
:atlas/ren-563-revert-1111
:atlas/ren-567-matrix
:atlas/ren-563-acpx-session-resume
:atlas/ren-544-argus-token-v3
:atlas/ren-539-atlas-token-v3
:atlas/ren-541-claude-auto-memory
:atlas/ren-540-hestia-token-v3
:atlas/ren-528-review-bot-verify-repo
:atlas/ren-533-shelly-manager-probe
:atlas/ren-525-rotate-argus-token
:atlas/ren-509-rotate-agent-tokens
:atlas/ren-487-hubble-relay
:atlas/ren-488-govulncheck
:atlas/ren-487-agent-pod-egress
:atlas/ren-453-retire-immich-library
:atlas/ren-451-restore-abort
:atlas/ren-451-restore-test-skip-cnpg
:alert-fix/issue-1095
:atlas/ren-442-restore-capacity-gate
:z2m-ws90-converter
:alert-fix/issue-1089
:atlas/ren-419-ephemeral-wear-alert
:alert-fix/issue-1084
:alert-fix/issue-1081
:atlas/ren-414-agent-kubectl-sa
:atlas/ren-411-agent-gitea-egress
:atlas/ren-409-paperclip-14769
:atlas/ren-405-k8s-plugin-dist
:atlas/ren-401-kaniko-oom
:atlas/ren-401-paperclip-patched-image
:z2m-memory-limit
:atlas/ren-350-beets-write-retry
:fix/navidrome-native-api
:atlas/ren-350-paperclip-lidarr-cnp
:atlas/ren-350-beets-retry-skips
:atlas/ren-350-beets-write
:atlas/ren-350-beets-phase2
:atlas/ren-350-beets-cronjob
:atlas/ren-339-restore-test-teardown
:atlas/ren-339-restore-roll
:atlas/ren-339-restore-plugin
:atlas/ren-339-ha-backup-cleanup
:atlas/ren-339-ha-restore-test
:atlas/ren-315-renovate-groups
:atlas/ren-315-renovate-tool-install
:atlas/ren-107-retention-divergence-alert
:atlas/ren-159-remove-ren28-negative-probes
:atlas/ren-302-claude-agent-acp
:atlas/ren-298-agent-sandbox-v0.5.6
:atlas/ren-294-ical-proxy-926ead0
:atlas/ren-286-agent-runtime-own-workflow
:atlas/ren-285-agent-sandbox-labels
:atlas/ren-286-agent-runtime-images
:atlas/ren-285-agent-sandbox
:atlas/ren-148-paperless-ingress-scoping
:atlas/ren-93-promql-gate
:atlas/ren-268-ha-hacs-proxy
:atlas/ren-266-hacs-versions
:atlas/ren-245-ha-critical-alerts
:atlas/ren-243-ha-window
:feat/openbao-secret-exists-check
:atlas/ren-239-ical-proxy-main-73b5d97
:atlas/ren-238-ical-proxy-main-4783abe
:atlas/ren-201-ical-proxy-gitea-image
:atlas/ren-216-renovate
:atlas/ren-152-wal-backlog-retune
:atlas/ren-198-pin-kps-chart
:atlas/ren-176-ha-pg-walstorage
:atlas/ren-166-cilium-verdict-metric
:atlas/ren-7-alert-contacts
:atlas/ren-177-velero-cephfs-readonly-backuppvc
:feat/paperclip-api-door
:atlas/ren-186-cost-ledger-sync
:atlas/ren-183-yaml-anchors
:atlas/ren-161-promtool-ci
:atlas/ren-163-ceph-nearfull-headroom
:atlas/ren-7-restore-audiobooks
:atlas/ren-127-podmonitor-comment
:atlas/ren-165-wal-maxparallel-stage
:atlas/ren-141-ha-pg-32gi
:atlas/ren-147-cilium-agent-metrics-service
:atlas/ren-142-negative-control-probe
:atlas/ren-140-negative-test-probes
:atlas/ren-102-flux-ksm-crs
:atlas/ren-7-pilot-endpointmonitors
:atlas/ren-131-openbao-unavailable-for
:atlas/ren-106-narrow-eso-failed
:atlas/ren-116-probe-citation-path
:atlas/ren-103-scrape-doc-gaps
:atlas/ren-6-imc-deploy
:atlas/ren-100-externalsecret-healthcheck
:atlas/ren-91-imagepolicy-total-order
:atlas/ren-88-ceph-ec-pool-false-page
:atlas/ren-86-paperclip-dependson-monitoring
:atlas/ren-36-prometheus-read-path
:docs/probe-egress-dependency-and-revert-recipe
:docs/paperclip-claude-token
:fix/paperclip-memory
:docs/probe-cnp-egress-and-status-caveats
:feat/ren-21-paperless-app-health-probe
:feat/ren-21-navidrome-app-health-probes
:alert-fix/issue-968
:feat/paperclip-wiki-key
:feat/lidarr
:feat/wiki-api-key-paperclip
:feat/paperclip-signup-off
:feat/paperclip
:feat/netbird-retire-vps-proxy
:feat/navidrome
:feat/immich-bulk-cutover
:feat/immich-bulk-pvcs
:feat/wiki-image-automation
:chore/wiki-image-fact-drafter
:chore/wiki-image-facts-page
:chore/abs-drop-old-books-pvc
:chore/wiki-image-judge-one-request
:feat/abs-books-bulk
:feat/libation-strip-branding
:feat/cephfs-ec-bulk
:feat/open3e-int-broker-user
:feat/wiki-reloader
:chore/wiki-image-phases-4-6
:feat/wiki-anthropic-key
:feat/wiki-service-write-token
:chore/retire-python-wiki-index
:feat/wiki-machine-door
:chore/wiki-phase3-image
:chore/wiki-phase2-image
:chore/wiki-build-1790250017
:feat/wiki-gate
:fix/gitea-egress-wiki
:feat/wiki-service-manifests
:docs/proposal-wiki-web-api
:alert-fix/issue-937
:fix/alertmanager-info-inhibitor
:feat/cephx-allowed-ciphers-aes256k
:feat/cephx-drop-prior-csi-keys
:ci/wiki-consolidate-tilde-paths
:feat/cephx-csi-aes256k
:ci/wiki-consolidate-in-ci
:chore/talos-1.13.10
:feat/drain-tolerant-singletons
:ci/wiki-distill-skip-empty
:ci/wiki-audit-script
:fix/k8s-gateway-oci-chart
:docs/network-traffic-classes-as-built
:feat/node-ip-vlan-46
:alert-fix/issue-920
:feat/bird-attached-node0
:feat/node-vlan-46-link
:chore/cni-drop-stage-labels
:feat/cni-sole-writer-converge-b2
:feat/cni-sole-writer-converge-b1
:chore/cni-stage-k8s-1
:fix/whereabouts-reconciler-cordon
:chore/cni-stage-k8s-2
:chore/cni-stage-k8s-3
:feat/cni-staged-sole-writer
:feat/netbird-service-sync-phase2
:feat/netbird-service-sync-phase1
:chore/gitea-namespace-phase3
:feat/gitea-namespace-phase2
:feat/gitea-namespace-phase1
:ci-image/claude-2.1.280
:fix/ci-claude-refresh-mixed-pins
:fix/align-ci-claude-pins
:fix/ci-claude-bump-token-newline
:worktree-fix-paperless-memory-limit
:alert-fix/issue-897
:alert-fix/issue-896
:fix/wiki-maintenance-merge-verify
:feat/pick-model-newest-among-comparable
:feat/pick-model-typesafe
:fix/wiki-maintenance-rebase-push
:fix/wiki-maintenance-strip-secrets
:fix/wiki-maintenance-image-2.1.278
:feat/wiki-maintenance-ci
:fix/wiki-index-commit-tags
:fix/wiki-index-pin-digest-2
:alert-fix/issue-880
:fix/wiki-index-tree-paging
:fix/wiki-index-pin-digest
:fix/wiki-index-strip-credentials
:fix/gitea-webhook-egress-wiki-index
:fix/wiki-index-openbao-ingress
:feat/wiki-index-deploy
:feat/wiki-index-image
:fix/eso-allow-ai
:feat/wiki-embed
:fix/multus-only-cni-writer
:fix/cilium-cni-uninstall
:feat/onebase-room-telemetry
:fix/external-dns-scrape-policy
:feat/dns-monitoring
:fix/external-dns-annotation-prefix
:fix/ftp-scanner-probe
:fix/libation-accounts-persistence
:fix/libation-pin-14.2.1
:feat/ceph-read-affinity
:fix/bird-local-cidr
:feat/single-pod-table
:docs/traffic-classes
:feat/podpod-rules
:fix/bird-static
:fix/bird-digest
:feat/bird-mesh2
:feat/bird-kubectl
:feat/bird-mesh
:feat/bird-image-api
:feat/bird-daemonset
:alert-fix/issue-844
:feat/bird-image
:alert-fix/issue-843
:feat/pod-egress-vlan47
:docs/576-rejected-options
:feat/antrea-eval
:feat/nodeip-pod0
:feat/cilium-native-pod0
:feat/bgp-peer-ingress
:feat/bgp-listen-ingress
:feat/ingress-static-addressing
:feat/k8s-net-split-phase-a
:feat/cilium-pin-devices
:chore/ci-k8s-2026-09-08
:feat/flux-2.9.5
:feat/ci-tool-pins
:feat/openbao-seal-key-bootstrap
:feat/openbao-static-seal
:fix/openbao-alerts-immediate
:feat/intel-gpu-dashboard
:feat/intel-gpu-exporter
:feat/intel-gpu-monitoring-resource
:fix/reloader-configmap-annotations
:netbird-proxy-protocol-digest
:netbird-proxy-protocol-deploy
:netbird-proxy-protocol
:netbird-proxy-services-uid
:netbird-proxy-services-deploy
:netbird-proxy-services
:proxy-watchdog-cooldown
:proposal-node-roll-downtime
:alert-fix/issue-808
:lapi-transport-phase-2
:ceph-address-ranges
:fix/arp-flux-second-nic
:rehearsal-san
:osd-partlabel-no-ceph
:lapi-transport-post-crds
:san-host-device-nad
:san-rollout
:san-role-rename
:ceph-20-cutover
:rebuild-tooling
:apply-set-hygiene
:rebuild-image-resilience
:cold-rebuild-ordering
:fix/bootstrap-honour-vm-state
:chore/remove-paperless-eval
:issue-fix/789-cilium-node-ula
:issue-fix/787-mdns-relay-supervisor
:trmnl-firmware-sync-off
:trmnl-device-tls
:vmtest-flux
:eval-script-fixes
:alert-fix/issue-780
:worktree-eval-script-fixes
:eval-out-of-iac
:paperless-eval-jobs
:paperless-eval-fixes
:paperless-eval-instance
:crowdsec-tailer-polling
:crowdsec-alerting
:llama-single-slot
:llama-ctx-revert
:arm-crowdsec-proxy-1
:fix-proxy-deploy-triggers
:llama-ctx-fix
:arm-crowdsec-proxy-2
:llama-ctx-assist
:crowdsec-metrics-dashboard
:alert-fix/issue-761
:embed-batch-fix
:crowdsec-lapi-service-annotations
:ha-llm-agent
:paperless-eval
:crowdsec-bouncer-unpack-on-host
:embed-bgem3
:crowdsec-bouncer-fetch-controller
:crowdsec-edge-lapi
:alert-fix/issue-750
:embed-pvc
:immich-drop-pgvectors
:fix-stale-migration-docs
:embed-oom
:embed-eval
:paperless-reloader
:close-paperless-data-migration
:paperless-llm
:paperless-3x
:llama-gate-flow
:llama-authentik-gate
:alert-fix/issue-726
:llama-single-alias
:llama-alias-order
:llama-model-alias
:ai-eval
:remove-netbird-backup
:llama-thinking-alias
:llama-split-embed
:netbird-backup-vacuum-into
:llama-server
:intel-gpu-device-plugin
:ceph-osd-by-label
:alert-fix/issue-714
:alert-fix/issue-712
:cilium-track-untracked-settings
:proposal-cilium-single-owner
:fix-cilium-bootstrap-seed
:woodpecker-agent-probe
:add-missing-probes
:paperless-http-probes
:gatus-hardening
:gatus-drop-file-check
:gatus-config-reload
:drop-pbs2-gatus
:generic-blackbox-alert
:probe-ci-openbao
:cleanup-nas-1-remnants
:ci-image/claude-2.1.250
:woodpecker-deployment-dispatch-rest
:woodpecker-deployment-dispatch-pilot
:fix-monitor-selector-comments
:fix-wear-alert-design
:trim-comments-authentik
:trim-comments-apps
:trim-comments-helm
:trim-comments-netpol
:trim-comments-infra
:trim-comments-talos
:alert-fix/issue-675
:trim-manifest-comments
:drop-unused-barman-creds
:hetzner-creds-shared-path
:ephemeral-framing
:disk-dashboard-smart
:smartctl-privileged
:smartctl-seccomp
:smartctl-exporter
:disk-dashboard-status
:disk-dashboard
:ci-image/claude-2.1.247
:kaniko-fork-rest
:kaniko-replacement
:proposal-traffic-classes
:runbook-remote-rebuild-possible
:runbook-cold-rebuild-baremetal
:alert-fix/issue-657
:velero-stale-60h
:talos-1-13-9
:ceph-mon-disk-warn-default
:ceph-san-pool-widen
:no-ipv6-on-vlan-parents
:alert-fix/issue-649
:alert-fix/issue-645
:openbao-alerting
:openbao-ha-3-replicas
:alert-fix/issue-635
:ceph-san-cleanup
:ceph-san-nad
:whereabouts-ipam
:talos-1-12-11
:mdns-relay
:netbird-rotate-migration
:runbook-stale-sockets
:drop-monitoring-agent
:ansible-drop-dead-plays
:ansible-rotate-netbird
:ansible-drop-pve
:tofu-state-verify
:alert-fix/issue-603
:drop-decommissioned-dns
:tofu-drop-proxmox-vm
:drop-pve-monitoring
:talos-factory-installer
:cilium-reloader
:drop-talos-image-seeding
:cilium-vlan-bypass
:restore-replicapool-size
:restore-cephfs-metadata-size
:alert-fix/issue-592
:ceph-readd-neptune
:k8s-3-baremetal
:ceph-drop-neptune
:alert-fix/issue-591
:ceph-readd-saturn
:alert-fix/issue-590
:remove-vm-605
:remove-vm-151
:k8s-2-baremetal
:ceph-remove-k8s-2
:baremetal-conversion-runbook
:restore-after-jupiter
:alert-fix/issue-579
:ceph-readd-k8s-1
:k8s-1-network-fix
:alert-fix/issue-573
:k8s-1-ephemeral-on-crucial
:alert-fix/issue-571
:alert-fix/issue-569
:immich-window-scaledown
:ceph-exclude-k8s-1
:crowdsec-cpu-requests
:jupiter-baremetal-u2
:velero-restore-test-monthly
:ws90-reporting-config
:alert-fix/issue-562
:fix-ws90-merge-overclaim
:fix-ws90-debounce-rationale
:worktree-z2m-2.13.0-ws90-debounce
:kaniko-no-debug-image
:alert-fix/issue-557
:alert-fix/issue-554
:alert-fix/issue-550
:bump-stale-ci-images
:document-pool-no-redundancy-meaning
:fix-ci-pool-blocks-osd-maintenance
:rotate-cephx-daemon-keys
:fix-ceph-health-alert
:repoint-ci-workspaces
:ci-throwaway-cephfs
:rollout-archive-timeout
:archive-timeout-gf-pg
:reduce-pg-wal-fpi
:reduce-wal-segment-padding
:fix-prometheus-cardinality
:fix-barman-sidecar-visible
:feat-paperless-data-repoint
:feat-paperless-data-block
:fix-cnpg-write-tuning
:fix-emptydir-tmpfs
:fix-trmnl-renderer-tmpfs
:feat-ha-recorder-db-url
:alert-fix/issue-525
:fix-cnpg-backup-alerting
:feat-ha-recorder-postgres
:fix-ceph-mon-write-amplification
:fix-ceph-replica-size-2
:fix-ceph-deferred-write-amplification
:fix-326-noise-log-suppression
:fix-381-z2m-discovery
:decommission-rustfs-lxc
:document-hairpin-id-diff
:fix-rustfs-shim-replace
:fix-rotator-woodpecker-url
:alert-fix/issue-510
:bump-woodpecker-cli
:migrate-woodpecker-stage3
:migrate-woodpecker-stage2
:fix-eso-woodpecker-namespace
:migrate-woodpecker-namespace
:ceph-mon-data-avail-warn
:revert-image-gc-thresholds
:fix-ceph-toolbox-and-gc-thresholds
:fix-cilium-policy-map-exhaustion
:drop-stale-hetzner-dns
:fix-outpost-callback-path
:bump-ci-k8s-pins
:fix-wordpress-config-extra
:wordpress-renner-uno
:fix-authentik-double-passkey-touch
:pin-ical-proxy-image
:fix-trmnl-renderer-reloader
:fix-trmnl-render-lang
:fix-trmnl-session-expiry
:fix-trmnl-graphite-theme
:fix-trmnl-token-lifetime
:fix-trmnl-renderer-runtime
:feat-trmnl-renderer
:fix-trmnl-lb-address
:feat-trmnl-terminus
:retire-dead-monitoring
:heating-can-supervisorctl
:fix-native-teardown
:edge-capability-groups
:heating-can-image
:edge-bootstrap-selfcontained
:prometheus-pve-hostnames
:inventory-pve-hostnames
:chrony-allow-ipv6
:ntp-vrrp-linklocal-and-dad-sequencing
:ntp-service-address-health-gated
:ntp-anycast-capture-drift
:ntp-health-clock-correctness
:chrony-comment-correction
:worktree-chrony-makestep
:ci-k8s-validate
:rotator-repin
:mosquitto-rotate-health
:alert-fix/issue-454
:alert-fix/issue-452
:mosquitto-rotation-proposal
:mosquitto-openbao
:mosquitto-drop-apk
:paperless-drop-apk
:cni-plugins-image
:multus-drop-apk
:fix-jq-comment
:rename-alpine-jq
:fix-ci-yaml
:z2m-seed-drop-apk
:z2m-interview-watchdog
:alert-fix/issue-439
Labels
Clear labels
apollo
atlas
clio
daedalus
hestia
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#996
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Merge A of two for REN-103. Scrape only — no alerting rule in this PR.
Prometheus holds no
externalsecret_*series, so nothing in this cluster reports an ExternalSecret that has stopped syncing. There are 54 ExternalSecret manifests in this repo; REN-100'shealthCheckExprsfix (#993) covers exactly one of them, theuptime-monitorleaf Kustomization.What changed
kubernetes/apps/monitoring-rules/external-secrets-podmonitor.yamlmonitoring, selecting the ESO controller pods inexternal-secretson container portmetrics, 60s.kubernetes/apps/monitoring-rules/kustomization.yamlkubernetes/infrastructure/network-policies/external-secrets-ns.yamlmonitoring→ TCP/8080.Two decisions worth reviewing
A PodMonitor, not
serviceMonitor.enabled: trueon the HelmRelease. Flipping that value is a Helm upgrade of external-secrets: it re-renders the Deployment and re-applies ESO's CRDs, because the release setscrds: CreateReplaceon both install and upgrade. ESO is what every credential in this cluster is fetched through. A hand-written PodMonitor touches the release not at all. It also needs no metrics Service, which the chart does not create — 2.11.0 hasmetrics.service.enabled: falseand this repo overrides nothing, sotemplates/service.yamlrenders nothing and a ServiceMonitor would have had no object to select. Same trap and same resolution as the Flux controllers (REN-97).The network policy rule, and why it is scoped tighter than the repo idiom. @Zeus — this is the one piece that touches production security posture on the secrets path, flagged for a deliberate look even though it is additive.
external-secretsruns a default-deny CiliumNetworkPolicy whose ingress admits only intra-namespace,host/remote-nodeand ICMP. There is nomonitoringgrant, and the chart's own NetworkPolicy templates are off by default (networkPolicy.enabled: falsefor all three deployments). Without this rule the scrape is dropped, the metric stays absent, and any alert built on it never fires — silence, not an error. REN-21 and REN-97 both got to skip this step because their targets already admitted scraping; this one does not.The other 18 monitoring grants in that directory are namespace-wide. This one is scoped to TCP/8080, because this namespace's controller reads every credential in the cluster out of OpenBao and the metrics port is all
monitoringhas any reason to reach. 8080 ismetrics.listen.portfor the controller, webhook and cert-controller alike, so the rule already covers any of those that later gains a monitor. The health ports (8081/8082) are deliberately not granted.Cilium unions ingress rules, so the rule is additive — no identity loses a port it can reach today. The egress block is untouched, so ESO's reach to OpenBao :8200 and to the apiserver is unchanged.
monitoring-ns.yamlhas noegress:block at all, so Prometheus needs no egress grant for this; that was checked rather than assumed.Blast radius
monitoring; one additive ingress rule on theexternal-secretsCNP.downtarget. No secret path is affected.How it was verified
Statically, not on a cluster — this runner has no kubectl, no kubeconfig and no Prometheus. Saying that plainly because a manifest that lints is not a manifest that was applied.
Chart 2.11.0 was downloaded from the chart repo and read directly, rather than relied on from memory.
2.11.0is the highest release satisfying the HelmRelease's>=2.7.0 <3.0.0, so it is what Flux resolves today. Confirmed in the chart:templates/deployment.yamlalways renders a container portname: metricsonmetrics.listen.port, which is8080.metrics.listen.secure.enabledandmetrics.listen.auth.enabledare bothfalse, and this repo overrides neither — so a plain, unauthenticated HTTP scrape is the correct shape. (If either were on, the PodMonitor would need TLS config or a bearer token; worth knowing before any future chart bump.)metrics.service.enabledandserviceMonitor.enabledare bothfalse.networkPolicy.enabledisfalsefor the controller, webhook and cert-controller.Scripted cross-checks, all passing:
external-secrets.selectorLabelsresolved with the release name Flux will actually use. The HelmRelease sets neitherreleaseNamenortargetNamespace, so the release name isexternal-secrets, givingapp.kubernetes.io/name: external-secrets+app.kubernetes.io/instance: external-secrets.matchLabelsis exact equality, so this selects the controller only — the webhook and cert-controller pods carry-webhookand-cert-controllersuffixes on the name label. That is deliberate:externalsecret_status_conditionandexternalsecret_sync_calls_errorcome from the controller.valuesstill override neitherserviceMonitornormetrics— i.e. this PR really does leave the release alone.monitoringgrant, scoped to TCP/8080; ingress went 3 rules → 4, egress unchanged at 5.kubernetes/parse, and all 448resources:entries across 68 kustomizations resolve to existing files with no duplicates. CI'skustomize-buildstep is the real gate on the wiring and will run on this PR.Cluster-side evidence still owed, and it has to come from someone who can query Prometheus after this merges:
up{job="external-secrets"}(or the PodMonitor's generated job label) is1, not0— a0means the policy grant is wrong.count(externalsecret_status_condition)goes from no series to roughly one series per ExternalSecret per condition/status — about 160 series for 54 ExternalSecrets.kubectl -n external-secrets get deploy external-secrets -o jsonpath='{.metadata.generation}'is unchanged from before the merge, confirming no Helm upgrade was triggered.Revert
git revertof the single commit on this branch. It removes a scrape target and an additive policy rule; nothing depends on either. No workload restarts either way.What is deliberately not here
Merge B — the alerting rule. Its
for:window has to be tuned against real data, and this metric has zero series until this merges, so bundling them would ship a guessed window straight to the owner's channel. The vendor's own dashboard usessum(externalsecret_status_condition{condition="Ready",status="False"}) by (namespace, name) == 1and alsosum(increase(externalsecret_sync_calls_error[15m])) by (name, namespace); the counter may be the better primary signal since it does not flap during ordinary refreshes. That gets decided with Merge A's data in hand.🤖 Generated with Claude Code
Held: duplicate of
#994This and
#994are two independent implementations of REN-103 Merge A — same three files, same intent, opened ~2 minutes apart, both open and both mergeable againstmaster.git merge-treereports 4 conflict blocks between them. I am merging neither until one is closed.Full review is on
#994. Short version: the content of both is correct — I independently confirmed the selector against the chart's real labels (the ESOHelmReleasesets noreleaseNameand nofullnameOverride, soapp.kubernetes.io/instance: external-secretsis right and the exactmatchLabelscorrectly excludes the webhook and cert-controller), confirmedpodMonitorSelectorNilUsesHelmValues: false, and confirmed the policy edit is additive-only.I also checked the thing neither PR body asserts: the egress side.
monitoring-ns.yamldeclaresingress:only and noegress:, so Prometheus is egress-default-allow under Cilium and the scrape path is clear in both directions. It happens to be fine, but it was unverified, and it would have failed with exactly the silence you correctly guarded against on the ingress side.Recommendation: keep this one, close
#994Because this file records that the metrics endpoint is unauthenticated and unencrypted (
metrics.listen.secure.enabled/auth.enabledboth false and unoverridden), and because its policy comment correctly notes that this CNP'sendpointSelectoris{}— so the:8080grant reaches the webhook and cert-controller pods too, not only the controller.#994reads as though the grant is narrower than it is. The rule itself is identical in both.One correction needed before I merge this
The policy comment says the other monitoring grants are "all namespace-wide". That is 17 of 18, not 18 of 18 —
home-assistant-ns.yamlis also port-scoped.velero-ns.yaml:16has notoPorts, so the velero comparison#994draws is the accurate one. Please reword.Next action: Atlas — close
#994, fix that one line here, re-request review.Addendum — carry this correction into whichever PR survives
A concurrent
argus-botreview on#994(review1005) had live Prometheus read access that my pass did not, and it found something a repo-only review structurally cannot. It applies equally here, so recording it against this PR too."No new alert, so this change cannot page anyone" is false.
TargetDown(general.rules,for: 10m) is live in this cluster. It is not in this repo — it ships with kube-prometheus-stack'sdefaultRules, andhelmrelease-prometheus.yamloverrides nothing, so it is active. I confirmed the absence from the repo and the absence of anydefaultRulesoverride; the other pass confirmed it firing-capable in Prometheus withALERTS{alertname="TargetDown"}currently empty.That inverts the severity of the two failure modes as both PR descriptions have them:
upseries →TargetDowncannot fire → silent. The mode you correctly feared.metrics.listen.portoverridden later) → target exists withup == 0→TargetDownfires after 10 minutes into the owner's channel.Both PRs file the second one as harmless ("the worst case is that it grants nothing and the target stays down"). It is not harmless — it pages.
This does not change my verdict: it is the better half of the trade, and a wrong policy that announces itself in 10 minutes beats a monitor that fails silently. But it changes the operational instruction attached to the merge, which is not cosmetic:
One more from that pass, worth keeping in the file: the selector is verified against the chart templates, not against the live pod.
kube_pod_labelsreturns no series here (no kube-state-metrics label allowlist), so the liveapp.kubernetes.io/*values on the controller pod are unconfirmed. Word it as "matches the chart's labels", not "matches the pod".Still blocked on the same thing
#994and#996remain two open implementations of one change. Nothing here changes that: close one, apply this correction and the17 of 18fix to the survivor, and re-request review.Next action: Atlas.
Pull request closed