Compare commits

...
Author SHA1 Message Date
claude-botandClaude Fable 5.1 67d506e1ac feat(ai): dedicated wiki-embed endpoint for the Notes ranker
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Second bge-m3 llama.cpp Deployment in ns ai so wiki queries never queue
behind Paperless on llama-embed. Shares the llama-server-models PVC, so
the weights are not downloaded again.

Exposed LAN-only as wiki-embed.renner.uno (no dns.renner.uno/public
label) for bin/wiki_rank.py on the workstation and the nightly wiki jobs.
The client is a script, and the Authentik gate is browser-only, so the
Ingress uses Traefik BasicAuth backed by an ExternalSecret, the same
shape as loki.renner.uno. A pod-scoped CiliumNetworkPolicy admits Traefik
to wiki-embed only; llama-server stays reachable solely via its gate.

Measured on 40 past wiki-synth queries: embeddings picking 150 candidates
plus one TypeSafe Choice puts a cited note in the top 5 for 95% of
queries at a sixth of the cost of ranking all 1700 notes with TypeSafe.

Needs before first use: OpenBao key apps/ai/wiki-embed-basic-auth with
field `users` (htpasswd line), and the same credential as user:password
in ~/.config/wiki-embed/basic-auth on the workstation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-19 23:23:30 +02:00
security-bot 932d07edbe chore(image-automation): bump ghcr.io/paperless-ngx/paperless-ngx:3.2.0
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-19 02:51:13 +00:00
security-bot 1a1205110d chore(image-automation): bump ghcr.io/home-assistant/home-assistant:2026.9.3
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-18 12:53:31 +00:00
security-bot aed827ae5b chore(image-automation): bump ghcr.io/music-assistant/server:2.10.4
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-18 09:51:06 +00:00
k.renner 3abcec5af7 Merge pull request 'feat(heating-can): read the One Base per-room telemetry over CAN' (#867) from feat/onebase-room-telemetry into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#867
2026-09-17 11:17:30 +00:00
k.renner 58444c9862 Merge pull request 'fix(external-dns): let Prometheus reach the metrics port' (#866) from fix/external-dns-scrape-policy into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#866
2026-09-17 08:35:13 +00:00
claude-botandClaude Opus 5 5f3aecb381 fix(external-dns): let Prometheus reach the metrics port
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The ServiceMonitor added in #865 discovers both external-dns targets and
then times out on them: the apps namespace is default-deny-ingress, so the
scrape never arrives.

  lastError: Get "http://10.244.2.218:7979/metrics": context deadline exceeded
  health: down

Same shape as gitea-allow-metrics-scrape, and one policy covers both
releases — the chart gives them the same app.kubernetes.io/name and they
share the namespace, even though external-dns-unifi is declared under
infrastructure/.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:29:53 +02:00
k.renner 4ad47b3925 Merge pull request 'feat(monitoring): watch DNS publication — external-dns metrics plus resolution probes' (#865) from feat/dns-monitoring into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#865
2026-09-17 08:26:04 +00:00
claude-botandClaude Opus 5 2131c7b5f5 feat(monitoring): probe that the LAN names still resolve
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
external-dns metrics say what the publisher intends. This says what a client
actually gets, and catches causes the publisher cannot see — a record removed
on the UDM directly, for one.

Two findings shaped it.

The probe asks the UDM directly rather than the cluster resolver. k8s-gateway
caches these names with `serve_stale 1h immediate`, so a deleted record would
keep answering from cache for an hour before the probe noticed.

And rcode alone is not a usable signal here. Measured against 10.2.40.1:

  does-not-exist-xyz.renner.uno            -> NOERROR, CNAME proxy.renner.uno
  does-not-exist-xyz.paperless.renner.uno  -> no answer

The public *.renner.uno wildcard matches one label deep, which is exactly
where ha, mqtt, ldap and syslog live: delete their records and the UDM still
answers NOERROR, pointing at the WAN proxy. A probe checking only the rcode
would have stayed green through the outage for four of the six names. So the
modules require an A record in the answer and reject a CNAME, which separates
a real record from the wildcard without encoding any address.

One module per name because the dns prober reads query_name from the module
and takes the resolver as its target; both families are probed per name and
the alert wants both to fail, so the UDM's bursty connection refusals do not
read as a missing record. That is also why the window is 30m rather than the
10m the HTTP probes use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:20:04 +02:00
claude-botandClaude Opus 5 d35c06d21d feat(monitoring): scrape external-dns and alert when it stops publishing
Neither external-dns release was scraped: both expose /metrics on :7979 and
no ServiceMonitor existed, so external_dns_* series were absent from
Prometheus entirely. The process that publishes every LAN name in the
cluster had no instrumentation at all.

That is how 2026-09-12 went unnoticed for five days. external-dns v0.22.0
changed the default annotation prefix, the service source went empty, and
policy=sync deleted six records. source_endpoints_total fell 48 -> 36 and
controller_verified_records 24 -> 18 per family; nothing was watching either.

ExternalDNSSourceEndpointsDropped is the rule for that shape: a sustained
drop below 90% of the 24h peak. The threshold is chosen against the observed
inventory — one retired Ingress is 2 endpoints of ~48 and stays quiet, a
whole class of names does not. A deliberate mass removal fires it once and
clears itself after 24h, which is the right trade for a signal that
otherwise only speaks when records are already gone.

The rest of the group covers the ways the drop rule itself can go blind:
source and registry errors, the instance being down, and an absent() dead
man mirroring BlackboxProbesAbsent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:15:18 +02:00
k.renner 1bbff9fca4 Merge pull request 'fix(dns): migrate external-dns hostname annotations off the alpha prefix' (#864) from fix/external-dns-annotation-prefix into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#864
2026-09-17 08:02:34 +00:00
claude-botandClaude Opus 5 9a521568c0 fix(dns): migrate external-dns hostname annotations off the alpha prefix
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
external-dns v0.22.0 changed the default --annotation-prefix from
external-dns.alpha.kubernetes.io/ to external-dns.kubernetes.io/ with no
fallback, and its release notes lead with the consequence: "This change can
delete all your DNS records."

The chart floated onto it at 2026-09-11 23:47:56 (appVersion 0.21.0 ->
0.22.0). Two hours later, at 2026-09-12 01:47:59, the unifi provider logged
`client deleted record` for every name that depended on the annotation:

  ha.renner.uno       mqtt.renner.uno     ldap.renner.uno
  syslog.renner.uno   ftp.paperless.renner.uno
  syncthing.paperless.renner.uno

A, AAAA and the TXT ownership records, and no `client created record` since.
Every one of those is a LoadBalancer Service, whose hostname external-dns
can only learn from the annotation — so the source went silent and
--policy=sync removed the records. Ingress-sourced names take their host
from spec.rules[].host and were untouched, which is exactly the split
observed: source_endpoints_total is 36 with source_errors_total 0.

These six annotations were the only alpha-prefixed external-dns annotations
in the repo, so moving them to the current prefix is the whole fix. Pinning
the prefix back with --annotation-prefix would keep us on a deprecated
default that upstream intends to drop.

Scanning has been broken since: the Brother resolves ftp.paperless.renner.uno
rather than using the address.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:00:30 +02:00
21 changed files with 613 additions and 10 deletions
@@ -40,7 +40,7 @@ metadata:
# so this Service produces no public record — LDAP stays
# LAN-internal, which matches the threat model of the cleartext
# bind on 389.
external-dns.alpha.kubernetes.io/hostname: ldap.renner.uno
external-dns.kubernetes.io/hostname: ldap.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -3,3 +3,4 @@ kind: Kustomization
resources:
- externalsecret.yaml
- helmrelease.yaml
- networkpolicy.yaml
@@ -0,0 +1,22 @@
# The apps namespace is default-deny-ingress, so the ServiceMonitor added in
# monitoring-rules/external-dns-servicemonitor.yaml scraped into a timeout.
# The selector matches both releases — the Cloudflare one deployed here and
# external-dns-unifi from infrastructure/ — since they share the chart's
# app.kubernetes.io/name and land in the same namespace.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: external-dns-allow-metrics-scrape
namespace: apps
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: external-dns
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: "7979"
protocol: TCP
@@ -18,7 +18,7 @@ metadata:
lb-pool: bgp
annotations:
lbipam.cilium.io/ips: "10.2.99.87,fd99::87"
external-dns.alpha.kubernetes.io/hostname: ha.renner.uno
external-dns.kubernetes.io/hostname: ha.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -95,7 +95,7 @@ spec:
memory: 32Mi
containers:
- name: home-assistant
image: ghcr.io/home-assistant/home-assistant:2026.9.2 # {"$imagepolicy": "flux-system:home-assistant"}
image: ghcr.io/home-assistant/home-assistant:2026.9.3 # {"$imagepolicy": "flux-system:home-assistant"}
env:
- name: TZ
value: Europe/Berlin
@@ -19,7 +19,7 @@ metadata:
# LAN can connect with `mqtt.renner.uno` (matches the broker cert
# SAN) instead of the bare IP. No Cloudflare label — kept off the
# public DNS; UniFi serves a split-DNS answer for LAN clients.
external-dns.alpha.kubernetes.io/hostname: mqtt.renner.uno
external-dns.kubernetes.io/hostname: mqtt.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -6,3 +6,5 @@ resources:
- embed.yaml
- gate-service.yaml
- ingress.yaml
- wiki-embed.yaml
- wiki-embed-ingress.yaml
@@ -0,0 +1,57 @@
---
# wiki-embed.renner.uno — LAN-only (no dns.renner.uno/public label), consumed by
# bin/wiki_rank.py on the workstation and the nightly wiki jobs. Machine client,
# so Traefik BasicAuth instead of the browser-only Authentik gate; same pattern
# as kubernetes/apps/monitoring/loki-ingress.yaml.
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: wiki-embed-basic-auth
namespace: ai
spec:
basicAuth:
secret: wiki-embed-basic-auth
realm: wiki-embed
---
# OpenBao key apps/ai/wiki-embed-basic-auth, field `users` = htpasswd line.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: wiki-embed-basic-auth
namespace: ai
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: wiki-embed-basic-auth
dataFrom:
- extract:
key: apps/ai/wiki-embed-basic-auth
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: wiki-embed
namespace: ai
annotations:
cert-manager.io/cluster-issuer: letsencrypt-cluster
traefik.ingress.kubernetes.io/router.middlewares: ai-wiki-embed-basic-auth@kubernetescrd
spec:
ingressClassName: traefik
tls:
- hosts:
- wiki-embed.renner.uno
secretName: wiki-embed-tls
rules:
- host: wiki-embed.renner.uno
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: wiki-embed
port:
number: 8080
@@ -0,0 +1,117 @@
---
# Embedding endpoint for the ~/Notes wiki ranker (bin/wiki_rank.py). Own
# Deployment so wiki traffic never queues behind Paperless on llama-embed.
# Same model and PVC as llama-embed, so no second download.
apiVersion: apps/v1
kind: Deployment
metadata:
name: wiki-embed
namespace: ai
labels:
app.kubernetes.io/name: wiki-embed
spec:
replicas: 1
# Above the 600s startupProbe budget; see embed.yaml.
progressDeadlineSeconds: 900
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: wiki-embed
template:
metadata:
labels:
app.kubernetes.io/name: wiki-embed
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: wiki-embed
image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
args:
- --hf-repo
- gpustack/bge-m3-GGUF:Q8_0
- --embeddings
- --alias
- bge-m3
# bge-m3 is trained for CLS pooling; the GGUF default is not.
- --pooling
- cls
- --n-gpu-layers
- "99"
- --threads
- "2"
# wiki_rank.py caps each note at 1500 chars and sends 2 per request.
- --batch-size
- "1536"
- --ubatch-size
- "1536"
- --ctx-size
- "4096"
- --no-webui
env:
- name: LLAMA_CACHE
value: /models
ports:
- name: http
containerPort: 8080
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
# Same sizing as llama-embed: 1.6 GiB measured peak, headroom for slots.
resources:
requests:
cpu: 250m
memory: 2Gi
gpu.intel.com/i915: "1"
limits:
cpu: "2"
memory: 8Gi
gpu.intel.com/i915: "1"
startupProbe:
httpGet:
path: /health
port: http
periodSeconds: 10
failureThreshold: 60
readinessProbe:
httpGet:
path: /health
port: http
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: http
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 6
volumeMounts:
- name: models
mountPath: /models
volumes:
- name: models
persistentVolumeClaim:
claimName: llama-server-models
---
apiVersion: v1
kind: Service
metadata:
name: wiki-embed
namespace: ai
labels:
app.kubernetes.io/name: wiki-embed
spec:
selector:
app.kubernetes.io/name: wiki-embed
ports:
- name: http
port: 8080
targetPort: http
@@ -0,0 +1,25 @@
# Both external-dns releases (Cloudflare and the UniFi webhook) carry
# app.kubernetes.io/name: external-dns and differ only by instance, so one
# monitor covers both. jobLabel keeps them apart as separate job values.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: external-dns
namespace: monitoring
labels:
app.kubernetes.io/name: external-dns
release: kube-prometheus-stack
spec:
jobLabel: app.kubernetes.io/instance
namespaceSelector:
matchNames:
- apps
selector:
matchLabels:
app.kubernetes.io/name: external-dns
endpoints:
# The controller's own metrics. Port 8080 on the unifi release is the
# webhook sidecar and exports no external_dns_* series.
- port: http
path: /metrics
interval: 60s
@@ -21,6 +21,14 @@ resources:
# alerted if that stopped — HetznerBackupEndpointDown only probes the S3
# endpoint, which proves reachability, not that anything was written.
- prometheusrule-cnpg-backup.yaml
# DNS publication. Nothing scraped external-dns, so it deleted six LAN
# records on 2026-09-12 and stayed silent for five days.
- prometheusrule-external-dns.yaml
- external-dns-servicemonitor.yaml
# Outcome check for the same failure: the names themselves, asked of the
# UDM. Catches a cause external-dns cannot see, e.g. records removed there.
- prometheusrule-dns-resolution.yaml
- probes-dns.yaml
- alloy-servicemonitor.yaml
- cilium-servicemonitor.yaml
- cnpg-podmonitors.yaml
@@ -0,0 +1,143 @@
# DNS probes for the names external-dns publishes to the UDM from
# LoadBalancer Services. One Probe per name because the dns prober's
# query_name is module-scoped; the target is the resolver, queried on both
# families so a single-stack blip does not read as a missing record.
#
# The UDM is queried directly rather than through the cluster resolver:
# k8s-gateway caches these with `serve_stale 1h immediate`, which would keep
# a deleted record green for an hour.
#
# `release` is load-bearing — probeSelector matches on it.
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-ha
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_ha
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
# instance is the resolver and identical across these Probes, so the
# queried name has to come from a label.
labels:
dns_name: ha.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-mqtt
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_mqtt
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: mqtt.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-ldap
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_ldap
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: ldap.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-syslog
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_syslog
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: syslog.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-ftp-scanner
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_ftp_scanner
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: ftp.paperless.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-syncthing
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_syncthing
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: syncthing.paperless.renner.uno
@@ -0,0 +1,49 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: dns-resolution
namespace: monitoring
labels:
# ruleSelector matches on this label; without it the rule is created but
# never loaded into Prometheus.
release: kube-prometheus-stack
spec:
groups:
- name: dns-resolution.rules
rules:
# min by dns_name: both resolver families are probed, and the record
# is only gone when neither answers. A single-family failure is the
# UDM's dnsmasq refusing connections in bursts (k8s-gateway carries
# the same note), which is why the window is 30m rather than the 10m
# used for HTTP probes.
- alert: LanDnsRecordMissing
expr: min by (dns_name) (probe_success{job="blackbox-dns"}) == 0
for: 30m
labels:
severity: critical
annotations:
summary: "{{ $labels.dns_name }} no longer resolves to its LAN address"
description: >-
The UDM stopped answering for a name external-dns publishes from
a LoadBalancer Service, or answered with the public
*.renner.uno wildcard instead of the record. Clients that use
the name rather than the address lose the service while
everything in the cluster stays healthy — nothing else in the
stack reports this. Check the external-dns alerts first, then
the record on the UDM.
# Dead-man's-switch, mirroring BlackboxProbesAbsent: without probe
# results the rule above cannot fire.
- alert: LanDnsProbesAbsent
expr: absent(probe_success{job="blackbox-dns"})
for: 15m
labels:
severity: critical
annotations:
summary: "No LAN DNS probe results are reaching Prometheus"
description: >-
The blackbox-dns Probes stopped being discovered or the exporter
rejects the modules. A 400 from /probe yields no probe_* series
at all, which looks the same as no Probe — check that every
dns_lan_* module the Probes reference exists in the
blackbox-exporter config.
@@ -0,0 +1,87 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: external-dns
namespace: monitoring
labels:
# ruleSelector matches on this label; without it the rule is created but
# never loaded into Prometheus.
release: kube-prometheus-stack
spec:
groups:
- name: external-dns.rules
rules:
# The signal that was missing on 2026-09-12: the sources stopped
# producing endpoints they had produced before, and policy=sync
# deleted the records. 10% below the 24h peak clears a single
# retired Ingress (2 of ~48) and catches a whole class of names.
- alert: ExternalDNSSourceEndpointsDropped
expr: >-
external_dns_source_endpoints_total
< 0.9 * max_over_time(external_dns_source_endpoints_total[24h])
for: 30m
labels:
severity: critical
annotations:
summary: "{{ $labels.job }} is publishing {{ $value }} endpoints, below its 24h peak"
description: >-
external-dns stopped seeing endpoints it had before. With
policy=sync the matching DNS records are deleted, so names stop
resolving without anything else failing. Compare
external_dns_source_endpoints_total against the last 24h, then
check whether the resources still carry a hostname annotation
the running version understands — a changed annotation prefix
default empties the source silently. Deliberate removals also
fire this and resolve themselves 24h later.
# Source errors mean the Kubernetes side failed; registry errors mean
# the provider did. Either way desired state is computed from partial
# input, which policy=sync acts on.
- alert: ExternalDNSSourceErrors
expr: increase(external_dns_source_errors_total[15m]) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }} is failing to read its sources"
description: >-
external-dns could not list the Kubernetes resources it derives
records from. Check the pod log and its RBAC.
- alert: ExternalDNSRegistryErrors
expr: increase(external_dns_registry_errors_total[15m]) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }} is failing to reach its DNS provider"
description: >-
external-dns could not read or write records at the provider.
For external-dns-unifi that is the UniFi webhook sidecar and the
controller behind it; for external-dns it is the Cloudflare API.
- alert: ExternalDNSDown
expr: up{job=~"external-dns.*"} == 0
for: 15m
labels:
severity: critical
annotations:
summary: "{{ $labels.job }} is not being scraped"
description: >-
The external-dns instance is down or unreachable. Records already
at the provider keep resolving, but nothing reconciles them any
more and every other alert in this group is blind.
# Dead-man's-switch for the group, mirroring BlackboxProbesAbsent: a
# dropped ServiceMonitor makes every rule above evaluate to nothing.
- alert: ExternalDNSMetricsMissing
expr: absent(external_dns_source_endpoints_total)
for: 15m
labels:
severity: critical
annotations:
summary: "No external-dns metrics are reaching Prometheus"
description: >-
Either both external-dns instances are gone or the ServiceMonitor
stopped being discovered. Verify it still carries
release=kube-prometheus-stack.
@@ -232,7 +232,7 @@ metadata:
lb-pool: bgp
annotations:
io.cilium/lb-ipam-ips: "10.2.99.84,fd99::84"
external-dns.alpha.kubernetes.io/hostname: syslog.renner.uno
external-dns.kubernetes.io/hostname: syslog.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -72,3 +72,73 @@ spec:
tcp:
preferred_ip_protocol: ip6
ip_protocol_fallback: true
# One module per name: the dns prober takes the RESOLVER as its
# target and reads query_name from the module, so a name cannot be
# passed per scrape. The Probes in monitoring-rules/probes-dns.yaml
# point these at the UDM.
#
# Both answer checks are load-bearing. valid_rcodes alone passes a
# deleted record two labels deep (NOERROR, no answer), and the public
# *.renner.uno wildcard answers one label deep with a CNAME to
# proxy.renner.uno — NOERROR with the WAN proxy behind it. Requiring
# an A and rejecting a CNAME separates a real record from both.
dns_lan_ha:
prober: dns
timeout: 5s
dns:
query_name: ha.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_mqtt:
prober: dns
timeout: 5s
dns:
query_name: mqtt.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_ldap:
prober: dns
timeout: 5s
dns:
query_name: ldap.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_syslog:
prober: dns
timeout: 5s
dns:
query_name: syslog.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_ftp_scanner:
prober: dns
timeout: 5s
dns:
query_name: ftp.paperless.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_syncthing:
prober: dns
timeout: 5s
dns:
query_name: syncthing.paperless.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
@@ -55,7 +55,7 @@ spec:
# stdlib only; Flux keeps the tag in lockstep via the shared image
# policy) — no extra dependency.
- name: reconcile-publish-ip
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.10.4 # {"$imagepolicy": "flux-system:music-assistant"}
command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
volumeMounts:
- name: data
@@ -65,7 +65,7 @@ spec:
readOnly: true
containers:
- name: music-assistant
image: ghcr.io/music-assistant/server:2.10.3 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.10.4 # {"$imagepolicy": "flux-system:music-assistant"}
env:
- name: LOG_LEVEL
value: info
+1 -1
View File
@@ -119,7 +119,7 @@ metadata:
# label — FTP must not be exposed via the cargo tunnel (clear-text
# protocol, public exposure not appropriate). The brother scanner
# uses this hostname instead of the LB IP.
external-dns.alpha.kubernetes.io/hostname: ftp.paperless.renner.uno
external-dns.kubernetes.io/hostname: ftp.paperless.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
+1 -1
View File
@@ -183,7 +183,7 @@ spec:
drop: ["ALL"]
containers:
- name: paperless
image: ghcr.io/paperless-ngx/paperless-ngx:3.1.3 # {"$imagepolicy": "flux-system:paperless-ngx"}
image: ghcr.io/paperless-ngx/paperless-ngx:3.2.0 # {"$imagepolicy": "flux-system:paperless-ngx"}
ports:
- name: http
containerPort: 8000
+1 -1
View File
@@ -145,7 +145,7 @@ metadata:
lb-pool: bgp
annotations:
io.cilium/lb-ipam-ips: "10.2.99.85,fd99::85"
external-dns.alpha.kubernetes.io/hostname: syncthing.paperless.renner.uno
external-dns.kubernetes.io/hostname: syncthing.paperless.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -126,3 +126,25 @@ spec:
- { type: 135, family: IPv6 }
- { type: 136, family: IPv6 }
- { type: 143, family: IPv6 }
---
# wiki-embed is the one ai pod Traefik may reach: it sits behind BasicAuth
# (apps/llama-server/wiki-embed-ingress.yaml), not behind the Authentik gate.
# Scoped to this pod so the default-deny above keeps Traefik off llama-server.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: wiki-embed-from-traefik
namespace: ai
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: wiki-embed
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
toPorts:
- ports:
- port: "8080"
protocol: TCP