Add a blackbox-http probe of kuma.renner.uno with a dedicated critical
UptimeKumaUnreachable rule (excluded from BlackboxProbeFailing), scrape
Kuma /metrics with the API key from OpenBao, and alert on sustained slow
monitor response times. No egress policy exists in the monitoring
namespace, so no CNP change is needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The flux-image-automation Alert forwarded every info event from
ImageUpdateAutomation (i.e. each successful bump commit) to Alertmanager,
which emailed it as FluxImageUpdateAutomationSucceeded. A success is not
a problem and must not be an alert; the commits are in git history.
Image-automation failures still alert: flux-failures covers
ImageUpdateAutomation at error severity. The Alertmanager source="flux"
route stays for those; its comment now says so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chart's KubeJobFailed fires while a failed Job object exists, so it
clears only on Job GC (TTL, history limit, Velero's keep-latest), not on
recovery. On 2026-10-18 this held 10 alerts open for runs that had already
been superseded: renovate recovered 2026-10-13 but alerted until its 7d
TTL; registry-bot-secret-rotate-init2 succeeded 9 min after init failed,
the alert started 7 min after that and stayed 11h.
Replace it with a rule that drops a failed Job once a later Job of the
same series completes: series = owning CronJob, or velero.io/repo-name
for kopia maintenance Jobs (exposed via KSM metricLabelsAllowlist).
Failed Jobs without a series alert as before.
KubeContainerWaiting fired for the never-started main container of a pod
whose init container failed. Exclude Failed/Succeeded pods; KubeJobFailed
covers those.
Verified against live Prometheus: the new expression returns nothing now
(chart rule: 2), returns the 30 Velero maintenance failures at 16:40Z
2026-10-18, and returns nothing at the renovate/registry-bot timestamps
after their successful re-runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama-server, llama-embed, wiki-embed (gpu.intel.com/i915) and the wiki
behind them are bound to the Intel iGPU. On a site without one (Hetzner,
restore rehearsal runs 27/28) their pods stayed Pending, and the GPU plugin
and exporter DaemonSets sat on every node with nothing to serve.
The node patches k8s-1..3 label their machines
hardware.renner.uno/intel-gpu=true. The GPU plugin, the exporter, the
three llama Deployments and the wiki now select on it. Elsewhere the
DaemonSets have no pods and the Deployments stay unscheduled.
Production rollout order: the label must be on the nodes (talosctl patch,
no reboot) before these manifests reconcile, or the GPU plugin and the
llama/wiki pods are removed or left Pending.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes the Paperclip deployment (namespace paperclip, CNPG cluster
pclp-pg, ingresses, image automation, server image build) and the
infrastructure that existed only for its agents:
- agent sandbox: paperclip-renner namespace and CNPs, the
kubernetes-sigs/agent-sandbox controller with its namespace, CNP and
Renovate rules
- agent-runtime-{go,ops} images and the build-agent-runtime pipeline
- matrix bridge (Element <-> Paperclip agents) and synapse's appservice
registration mount
- monitoring loki-read-gateway (only called by the paperclip Role)
- MQTT account agents-read and the controller's RBAC in ns paperclip
- the OpenBao secret-exists-check policy file
- agent admits in the wiki, lidarr, shelly-manager, mosquitto and
hubble-relay CNPs; the paperclip-api Gitea webhook host; the wiki API
key; the ClusterSecretStore namespace entry; the pclp-pg RECOVERY row
- the Paperclip design spec and plan
Authentik group "Paperclip Users" is switched to state: absent so the
blueprint deletes it. package-retention's EXCLUDE_PACKAGES becomes
optional, since its only value was the agent-runtime prefix.
The shelly-manager api-proxy sidecar stays: mqtt-account-controller
calls shelly-manager-api too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 of unifying the auth flows, after the llama and paperless trial
of stage 1 was confirmed:
- Brand authentik.srv.renner.uno: flow_authentication =
sso-authentication (was NULL, i.e. the slug-first
default-authentication-flow with password + MFA). Every provider
without its own authentication_flow logs in through it, PVE and PBS
included.
- The 7 blueprint OIDC providers (Home Assistant, NetBird,
Audiobookshelf, Immich, Gitea, Grafana, Matrix MAS): authentication
flow unset (brand), authorization implicit consent.
- authentik-gate-sync: default authorization flow implicit consent, via
DEFAULT_AUTHORIZATION_FLOW on the Deployment and in the code; the
per-gate login/authorization annotations are dropped. Paperless keeps
paperless-authorization.
Every flow referenced here already exists, so the gate controller does
not hit the missing-flow removal seen in stage 1. Stage 3 removes the
passkey-only flows once nothing references them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
paperless, shelly-manager, zone-configurator, esphome, zigbee2mqtt,
alertmanager, prometheus, wiki, paperclip, navidrome, lidarr and wordpress
now gate through authentik-gate-sync: each source Ingress names its real
backend and carries the old provider's settings as annotations (group,
login-flow, authorization-flow where it was implicit consent, paperless'
skip-path).
Removed per gate: the ExternalName Service, the egress CNP in authentik,
the namespace CNP rule admitting the old outpost, and the blueprint
provider/application/binding/outpost (state: absent, outpost ->
application -> provider). Groups stay declared without users lists;
membership is managed in Authentik.
Per-gate specifics:
- paperless: the docs.renner.uno alias backends the generated
paperless-gate Service (its middleware redirects before the backend); the
auth.renner.uno/outpost.goauthentik.io route is removed, proxy outposts
take their callback on the app host.
- alertmanager / prometheus: source Ingresses renamed to alertmanager /
prometheus; the controller generates the *-gate route Ingresses.
- wordpress-admin: gains a tls block for www.renner.uno (wordpress-tls,
certificate still owned by the public Ingress), so the gate's external
host stays https.
- wiki, paperclip: only the gate peer/rule leaves their shared CNPs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The udm_cef regex expected UNIFIhost immediately after the CEF header and
UNIFIdeviceName/-Model/-Ip/-Mac/-Version before msg. The UDM emits
UNIFIcategory first and the extension keys differ per event type, so no line
matched and no labels were promoted.
Anchor on the header, UNIFIcategory, UNIFIhost and the trailing msg, and
promote vendor, product, signature_id, event_name and category.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Loki runs SingleBinary with auth disabled and answers GET on
/ingester/shutdown and /flush, so an RBAC `get` on services/proxy for
loki:3100 would not be read-only. Put an nginx allow-list in front of it
(query, query_range, labels, label/<name>/values, series; GET only;
upstream URI rebuilt per location) and grant the paperclip read SA
`get` on that Service only. Owner decision: REN-622 (option A2).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision on REN-107: option B, 30 days on the existing disk.
retention 90d -> 30d, retentionSize 30GB -> 40GiB, PVC unchanged (50Gi).
At ~1 GB/day the 30GiB cap bound first and the real window was a
20.5-27.2 day saw-tooth; 30d peaks near 38 GB, under the 40GiB backstop.
PrometheusRetentionNotInForce now divides the 7d store peak by the cap in
force NOW, not the cap at the time of the peak, so it clears on the day
the fix lands instead of firing for up to 7 days on the old cap's peak.
Adds tests/prometheus-retention_test.yaml (5 scenarios) and 5
`retention-*` mutations, all caught.
Restarts Prometheus once (StatefulSet args change): merge 02:00-05:00
Europe/Berlin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The version range >=72.0.0 let Flux upgrade the chart unattended
(91.5.2 -> 91.7.0 at 19:43 UTC on 2026-09-26, no git commit). That
restart moved Prometheus onto k8s-1 while its Ceph RBD mounts were
broken, causing a ~48 min alerting outage (REN-198).
Pinned to the version already deployed, so reconcile is a no-op.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments that pointed at other YAML as `file.yaml:N` went stale on any
edit above the cited line; three drift fixes had been merged by hand
(REN-181). The cited block now carries `# anchor: <name>` and the
comment cites `path.yaml#<name>`.
- Sweep: all 32 line-number citations in YAML comments converted.
Several were already stale at head and now point at the right block
(see PR). Vendored files (Flux gotk-components, IMC CRD) get no
anchor; those two citations name the object/schema path instead.
- Guard: scripts/ci/yaml-citations.sh in ci-manifests-test.yml; fails
on a line-number citation or a path#name whose file/anchor is gone.
Self-test in ci-scripts-test.yml.
- CONTRIBUTING.md: the convention in three lines.
Comment-only under kubernetes/: every file parses to identical objects
before and after, so Flux applies nothing and no ConfigMap data changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-object Flux readiness cannot come from the controllers: gotk_reconcile_
condition and gotk_suspend_status are declared by fluxcd/pkg/runtime and
populated by nothing at the pinned tags, so the REN-97 scrape yields 2,273
gotk_* series and none of them are about readiness. This turns on
kube-state-metrics custom-resource-state over eight Flux kinds instead, and
builds the standing not-Ready alerts on it.
The not-Ready half is complete. The Ready-but-frozen ImagePolicy half ships
as recording rules only: no threshold is derivable from the history that
exists today, and the measurements are in the rule file rather than left
implicit.
- kube-state-metrics customResourceState + rbac.extraRules (list, watch only)
- prometheusrule-flux.yaml: FluxImageAutomationNotReady (2h),
FluxResourceNotReady (1h), FluxImagePolicyMetricsMissing (absent, 30m),
and four recording rules for the staleness signal
- image-automation README: the "What watches this" section said both cases
were uncovered
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Our hand-written Alertmanager config replaced the chart's default and
dropped two pieces kube-prometheus relies on:
- a route sending InfoInhibitor to the null receiver
- the inhibit rules that let InfoInhibitor (and warning/critical)
suppress severity=info alerts
So both the helper alert and the info alerts it exists to silence were
emailed, daily. InfoInhibitor fired over the last 7 days in crowdsec,
kube-system, woodpecker-ci and without a namespace, driven by
CPUThrottlingHigh (crowdsec-agent, whereabouts-reconciler),
KubeletTooManyPods and KubeQuota* in woodpecker-ci.
Route and the three inhibit rules are copied from kube-prometheus
manifests/alertmanager-secret.yaml. Info alerts now reach email only
while a warning or critical alert fires in the same namespace, which is
the upstream design. amtool check-config passes; amtool config routes
test sends InfoInhibitor to null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kubernetes/apps/gitea: every object in namespace gitea; applied by a
new gitea Flux Kustomization (clusters/homelab/gitea.yaml) instead of
apps. gitea-allow-metrics-scrape dropped: the gitea-ns baseline
already admits monitoring.
- Rotators (flux-git, review-bot) and the Prometheus gitea job use
gitea.gitea.svc.
- bootstrap.sh waits for the gitea KS.
Data: data-gitea-0 is not copied. The PV (reclaimPolicy Retain) is
rebound to a PVC gitea/data-gitea-0 pre-created with volumeName, which
the new StatefulSet adopts. gitea-tls is pre-issued in the gitea
namespace so HTTPS has no certificate gap.
Refs #878
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
external-dns metrics say what the publisher intends. This says what a client
actually gets, and catches causes the publisher cannot see — a record removed
on the UDM directly, for one.
Two findings shaped it.
The probe asks the UDM directly rather than the cluster resolver. k8s-gateway
caches these names with `serve_stale 1h immediate`, so a deleted record would
keep answering from cache for an hour before the probe noticed.
And rcode alone is not a usable signal here. Measured against 10.2.40.1:
does-not-exist-xyz.renner.uno -> NOERROR, CNAME proxy.renner.uno
does-not-exist-xyz.paperless.renner.uno -> no answer
The public *.renner.uno wildcard matches one label deep, which is exactly
where ha, mqtt, ldap and syslog live: delete their records and the UDM still
answers NOERROR, pointing at the WAN proxy. A probe checking only the rcode
would have stayed green through the outage for four of the six names. So the
modules require an A record in the answer and reject a CNAME, which separates
a real record from the wildcard without encoding any address.
One module per name because the dns prober reads query_name from the module
and takes the resolver as its target; both families are probed per name and
the alert wants both to fail, so the UDM's bursty connection refusals do not
read as a missing record. That is also why the window is 30m rather than the
10m the HTTP probes use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
external-dns v0.22.0 changed the default --annotation-prefix from
external-dns.alpha.kubernetes.io/ to external-dns.kubernetes.io/ with no
fallback, and its release notes lead with the consequence: "This change can
delete all your DNS records."
The chart floated onto it at 2026-09-11 23:47:56 (appVersion 0.21.0 ->
0.22.0). Two hours later, at 2026-09-12 01:47:59, the unifi provider logged
`client deleted record` for every name that depended on the annotation:
ha.renner.uno mqtt.renner.uno ldap.renner.uno
syslog.renner.uno ftp.paperless.renner.uno
syncthing.paperless.renner.uno
A, AAAA and the TXT ownership records, and no `client created record` since.
Every one of those is a LoadBalancer Service, whose hostname external-dns
can only learn from the annotation — so the source went silent and
--policy=sync removed the records. Ingress-sourced names take their host
from spec.rules[].host and were untouched, which is exactly the split
observed: source_endpoints_total is 36 with source_errors_total 0.
These six annotations were the only alpha-prefixed external-dns annotations
in the repo, so moving them to the current prefix is the whole fix. Pinning
the prefix back with --annotation-prefix would keep us on a deprecated
default that upstream intends to drop.
Scanning has been broken since: the Brother resolves ftp.paperless.renner.uno
rather than using the address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Grafana dashboard for the gpumon_* series from intel-gpu-exporter:
current Render/3D busy, GPU power and GT frequency per node as stat
tiles, then one time series per engine (Render/3D, Video, VideoEnhance,
Blitter) with node as the series, GT frequency, GPU and package power,
and memory-controller read/write bandwidth.
Node colours are fixed per node across every panel — blue, orange, aqua
for k8s-1/2/3 — rather than assigned by series order, so a node keeps
its colour whatever the $node filter shows. The trio validates for
colour-vision deficiency and normal vision on both Grafana themes
(worst all-pairs ΔE 9.4 / 20.9). Reads and writes share one axis and
differ by dash, not hue. One axis per panel throughout.
gpumon_clients_count is deliberately absent: per-client attribution
needs hostPID, which the exporter does not have, so it would read 0
forever and imply an idle GPU.
Refs #752
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The -device flag named /dev/dri/card0. That is the right node today on
every host, but nothing guarantees the number: a second GPU or a
different probe order and the exporter would silently watch the wrong
device or none. The plugin's monitoring resource injects every GPU on
the node and intel_gpu_top selects it on its own — verified with a probe
pod on k8s-2, which produced the same gpumon_* series without the flag.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A DaemonSet running clambin/intel-gpu-exporter (intel_gpu_top behind a
Prometheus endpoint) with a ServiceMonitor, giving gpumon_engine_usage,
gpumon_frequency, gpumon_power, gpumon_imc_* and gpumon_clients_count
per node.
Privilege model, validated with a one-shot pod before this commit:
/dev/dri comes from the device plugin's gpu.intel.com/monitoring
resource (every GPU on the node, no hostPath); the i915 PMU that
intel_gpu_top reads is a system-wide event source and the nodes run
perf_event_paranoid=3, so CAP_PERFMON is the one capability added.
No privileged, no hostPID, no hostNetwork, read-only root filesystem.
Root rather than a fixed uid because the capability has to be effective
without file capabilities on the binary.
End-to-end check on k8s-2 during a 40 s chat completion on llama-server:
Render/3D busy 98 %, actual frequency 1449 MHz, GPU power 18.8 W, while
the pod's CPU stayed near 100m — the "is it really on the GPU" question
this exists to answer.
XPU Manager was ruled out (Arc Pro / data-centre parts only); the
archived onedr0p exporter needs privileged + hostPID.
Fixes#752
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A dashboard first needed something to draw. Nothing scraped CrowdSec:
`up{namespace="crowdsec"}` returned zero series and Prometheus held no cs_*
metric at all, so the detection half of the ingress path was invisible —
including whether the agents were reading logs and whether any bouncer was
pulling decisions.
Both chart services already expose a `metrics` port on 6060 and the
crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
without a port restriction. Only the ServiceMonitor was missing. One object
covers both services via matchExpressions, since they differ only in `app`.
Every panel query was written against label sets read off the live endpoints
rather than from the documentation:
cs_active_decisions{action,origin,reason}
cs_lapi_bouncer_requests_total{bouncer,method,route}
cs_bucket_overflowed_total{name}
cs_filesource_hits_total{acquis_type,datasource_type,source}
cs_node_wl_hits_ok_total{...,reason,...}
cs_parser_hits_ok_total / _ko_total
The bouncer panel is the one worth having: the LAPI counts requests per
bouncer, so a bouncer that stops pulling shows up as a line going to zero
rather than as nothing at all. It also makes the `<name>@<ip>` behaviour
visible — the LAPI registers a new entry per observed client address, and what
it observes is the netbird-ingress pod IP, so one logical bouncer appears
several times and gains new names as those pods roll.
The last panel is a text panel naming what the dashboard cannot show. The
bouncers' own metrics — fw_bouncer_active_decisions, dropped packets, their
view of the LAPI — sit on 127.0.0.1:60601 on public hosts with no metrics path
of any kind. So "Bouncer-Abholrate" proves a bouncer is asking, not that it
applied the answer. Better to say that on the dashboard than to let someone
read enforcement into a panel that does not measure it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Chasing a stale comment turned up a live duplicate. Since serviceMonitorSelector
was loosened to {}, Prometheus discovers the goauthentik.io operator's own
ServiceMonitors — which is why it was loosened. The hand-written copies for ldap
and paperless-gate, added back when the strict selector was dropping the
operator's, now scrape the same endpoints a second time.
Measured, not assumed: 10.244.4.113:9300 and 10.244.4.76:9300 were each
producing 40 samples per 10 minutes against 20 for every other outpost, and
ldap picked up a third target because the hand-written selector matched the
plain Service as well as the -metrics one. No duplicate-timestamp errors,
because the two pools scrape at different offsets — the cost is doubled
ingestion and a target list that lies about how many outposts there are.
Radius stays, and the comment now says why it is not the same case: the
operator's ServiceMonitor for it selects on managed-by=goauthentik.io together
with service-type=metrics, and no Service carries both — the operator's has the
first, the one defined here has the second. It is the only thing scraping
radius, so deleting it would have been a silent loss.
Also corrects the comments that claimed the release label is required for
discovery. It is not, for ServiceMonitors and PodMonitors: those two selectors
are empty. It very much is for Probes, PrometheusRules and ScrapeConfigs, whose
selectors still match on it, so probes-blackbox.yaml now says so explicitly —
otherwise the corrected comments elsewhere read as license to drop the label
there too.
Live selector state, for the record:
serviceMonitorSelector {} podMonitorSelector {}
probeSelector / ruleSelector / scrapeConfigSelector release=kube-prometheus-stack
Config files were carrying investigation write-ups: two ruled-out hypotheses in
the smartctl-exporter securityContext, a twenty-line canonical note on the
Hetzner credential, the history of an alert threshold. That is derivation, not
configuration, and it belongs where it does not rot — the commit message for how
we got here, the wiki for what holds.
The distinction this applies: a comment stays if it changes what someone editing
that line would do — a constraint of the tool, a trap, a pointer to an issue. It
goes if it narrates how a conclusion was reached, restates a measurement, or
justifies a value after the fact. The test is whether the surrounding config
could change and leave the comment quietly wrong.
160 lines out, 30 in. Nothing behavioural changed; both manifests still pass a
server-side dry run.
Alert descriptions were tightened rather than removed. Those are not comments —
they are what someone reads when paged, and an essay is worse there than
anywhere.
This covers only what I added today. The repo has roughly 8600 comment lines
with 88 files over 25, and going further needs the criterion above agreed
first — much of that volume is load-bearing warning rather than narration.
I described the worn drives carrying EPHEMERAL as the dangerous placement,
repeatedly, in the alert text and in the dashboard's caveats. That reads the
volume backwards. EPHEMERAL is Talos's /var and its contents are meant to be
disposable — container images, kubelet state, pod scratch, spegel and Rook's
local state all rebuild. Putting the drives with the least endurance left there
is a sensible destination, not a risk someone should be warned about.
The alert said "a node simply stops when it goes", which is simply wrong. A node
losing /var rebuilds it.
What survives the correction is narrower and checked rather than recalled:
/var/lib/etcd is on that volume too, so losing EPHEMERAL on a control-plane node
costs that member's data. Quorum holds at two of three and the member is removed
and re-added. That is a rebuild, not data loss, and the alert now says so at
that weight.
The fill-level panel keeps its warning but for the right reason: a full /var
stops a node being useful long before anything is lost, and nothing rebalances
it the way Ceph rebalances a full OSD. That is a capacity argument, not an
endurance one.
None of this changes the monitoring — measuring every drive is still right, and
the wear figures are unchanged. It changes what a reader is told the numbers
mean, which is the part that was misleading.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The exporter reports all nine drives now, so the dashboard shows what it
measures: a worst-wear stat in the status row, wear per drive over time, and a
table of the failure-now signals — available spare, media errors, the critical
warning bitfield, power-on time and written bytes.
Every expression was run against the live TSDB and the label names read off real
series rather than a README. That mattered: the exporter labels devices `nvme0`
and carries model and serial on a separate `smartctl_device` info metric, so
each query joins on (node, device) to pull the model in. A legend on the device
name alone would compare different hardware across machines, which is the same
trap the block-layer panels already avoid.
Wear and damage are presented as different things because they are. The wear
percentage is a budget the drive spends and says nothing about tomorrow; spare
exhaustion, media errors and the critical warning mean something is wrong now.
The caveats panel says so, and no longer claims endurance is missing — it was
missing when that text was written and is not any more.
Also fixes the OSD stat, which still rendered two identical numbers side by
side. The commit that fixed it did not make it into #668 before that PR merged,
so the panel went out in the state that prompted the complaint. It now shows
OSDs down alone; up-but-out is a state rather than a fault and Ceph warns about
it separately.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third attempt, and the first two were wrong for the same reason: I kept looking
for the gate in the capability and syscall layers, and it was never there.
Attempt one gave the container CAP_SYS_ADMIN and CAP_SYS_RAWIO with everything
else dropped. It found all three drives per node and could open none of them.
Attempt two added seccompProfile: Unconfined, reasoning that EPERM is the
default action for a syscall an unlisted profile blocks and that the kubelet
runs seccompDefault: true. /proc/1/status confirmed both capabilities effective
and a filter loaded, so the theory was at least testable — and removing the
filter changed nothing.
What settled it was a smaller test than either: `head -c 512 /dev/nvme0n1`
inside the container also fails with EPERM. Reading a block device involves no
ioctl, no capability and nothing a seccomp profile filters. The denial is the
device cgroup, which containerd applies to every non-privileged container
regardless of whether /dev is mounted. Mounting /dev makes the nodes visible;
it does not make them usable.
Kubernetes has no per-device allowlist in a Pod spec — no equivalent of
`docker --device`. So the honest position is that `privileged: true` here is
not a lazy shortcut standing in for a narrower setting; it is the only
mechanism the API offers, short of writing a device plugin. The manifest now
says that, and records what was ruled out and how, so the next reader does not
spend the same two rounds in the capability layer.
The /dev mount loses its readOnly flag with it. smartmontools opens NVMe
devices read-write on some paths, and once the container is privileged that
flag suggests a boundary that is not there.
A device plugin advertising the drives as a schedulable resource is the
genuinely narrower answer and is noted as such. It is worth building when more
than one workload needs raw device access; for a single read-only metrics
exporter it is a component to write, deploy and maintain in exchange for the
same access.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The exporter deployed cleanly, ran 3/3 with no restarts, answered its liveness
probe, and produced no drive metrics at all. Only smartctl_devices and
smartctl_exporter_build_info reached Prometheus — a green DaemonSet measuring
nothing, which is the failure mode the dashboard's own caveats panel warns
about.
The logs said why: it found all three drives per node and could open none of
them, with 'Smartctl open device: /dev/nvme0 failed: Operation not permitted'.
EPERM rather than EACCES is the tell — it is the default seccomp action for a
syscall a profile does not list. Rather than guess between that and the
capabilities, I read /proc/1/status in the running container: CapEff was
0x220000, so SYS_ADMIN and SYS_RAWIO were both effective, while Seccomp was 2
with one filter loaded. The capabilities were never the problem. The kubelet
runs with seccompDefault: true, so every pod that does not say otherwise gets
RuntimeDefault.
SELinux was ruled out the same way: /sys/fs/selinux/enforce reads 0.
This stays narrower than privileged: true, which would additionally grant every
capability, all host devices and write access to /sys. The tighter option is a
custom seccomp profile allowing exactly the ioctl smartctl issues; that is worth
building if this set of exceptions ever grows, and is not worth it for one pod.
Refs #549
Drive wear has been invisible in every signal this cluster has. `ceph -s` reads
active+clean at any wear level, no Ceph health check fires until a device
actually fails, and the block-layer metrics show throughput and latency but
never endurance. That blind spot is how a set of OSD drives reached 81-87%
before anyone looked (#549, #434).
This closes it for every drive, and it costs a deliberate exception to the
no-privileged-containers rule. The owner made that call explicitly after being
shown the alternative.
The alternative was real and is worth recording. Ceph's devicehealth module
already archives complete smartctl output daily — percentage_used,
power_on_hours, media_errors, critical_warning, available_spare — collected
without any elevated privileges, because the OSD pods hold the device access
already. Exporting that needs no exception at all. It also covers OSDs and
nothing else: three of nine drives here, and the six it misses include the ones
retired from Ceph for wear that now carry EPHEMERAL. Monitoring the healthy
drives while leaving the worn ones dark was not worth a new service.
The requirement comes from the kernel, not from the tool. NVMe health is read
through an admin ioctl on the character device, which is gated on CAP_SYS_ADMIN;
sysfs publishes the controller temperature and nothing more. Swapping
smartctl_exporter for another exporter would not change that, so the header says
so — the thing to re-check one day is whether the kernel grew a sysfs interface,
not whether some other tool asks for less.
The exception is kept to the smallest form that works: no `privileged: true`, no
host namespaces, no hostPID. Root plus CAP_SYS_ADMIN and CAP_SYS_RAWIO, all
other capabilities dropped, a read-only root filesystem, and /dev as the only
host mount — required because Kubernetes cannot hand individual character
devices to a pod without a device plugin. The admission policy that forbids
privileged pods is bound to the woodpecker-ci namespace only, so nothing here
circumvents it; monitoring already enforces pod-security `privileged`.
Two details that would otherwise have made the metrics useless:
The ServiceMonitor relabels the pod's node onto every series. Without it each
sample carries only the DaemonSet pod's IP, which says nothing about which
machine the drive is in and changes on every reschedule — an unattributable wear
figure is most of the value gone.
No --smartctl.device list. The exporter scans and rescans on its own, and a
written-down device list would quietly stop covering a drive the next time one
is replaced. Device names are not stable across these machines in the first
place.
The alert rules deliberately separate wear from damage: a wear percentage is a
budget being spent and warns at 80, while spare exhaustion, a critical-warning
flag or new media errors mean the drive is failing now. Media errors alert on
the increase rather than the total, so a drive that recorded errors long ago
does not alert forever while one that starts failing today does.
The completeness check compares the number of reporting drives against
node-exporter's own NVMe inventory rather than a written-down fleet size, so
adding or removing a drive moves both sides together. A rule that alerts on
nothing looks exactly like a healthy cluster, which is the failure mode worth
guarding.
Dashboard panels follow separately, once the metrics are live and the label
names can be read off the real series rather than guessed from a README.
Closes#549
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The descriptions I wrote listed the drives fitted today: which model holds which
role, that nvme0 is the system disk on two named nodes and something else on the
third, that a count below three means a drive is gone. All true this afternoon,
none of it true after the next swap — and prose does not get re-read when
hardware changes. That is the same failure this session already found in the
velero schedule comment, in the ci-claude consumer list, and in an instruction
pointing at a deleted file. I built a fresh instance of it while writing about
the others.
The text now describes mechanisms that outlive any particular set of drives, and
points at the inventory table for anything specific. Roles are assigned by serial
in Talos and by by-id path in Rook; a broad `disk.transport == 'nvme'` selector
matches every drive in the box; a drive retired from Ceph is not necessarily out
of service. None of that changes when the hardware does.
The OSD-count panel had the same problem in a worse form, since it was a
threshold rather than a sentence: green at exactly 3. It now reads OSDs down and
out as a difference against ceph_osd_metadata, so growing or shrinking the
cluster does not turn the panel red.
Fixes a claim that was simply false: the inventory description said the OSD
column came from ceph_disk_occupation, and the query never joined it. There was
no such column. It exists now, as a PromQL outer join — the matched half via
group_left, the unmatched half via `unless`, unioned — so drives with no OSD
still appear. Verified against the live TSDB: nine rows, three carrying a
ceph_daemon.
A Grafana `merge` transformation would have been the shorter route and would not
have worked. Merge only combines rows whose shared fields all agree, and the two
queries disagree on `instance` and `job` — node-exporter reports an IP and port
where ceph-mgr reports a node name — so it would have produced twelve rows
instead of nine, silently. Doing the join in PromQL also made it testable before
commit rather than after deploy.
Temperature thresholds are now labelled as generic NVMe limits rather than any
drive's rating, since they are not derived from a datasheet.
Refs #549
The first version graphed throughput, latency and temperature but never answered
"is anything wrong". Establishing that still meant querying Prometheus and Ceph
by hand — which is exactly the work a dashboard exists to remove, and which the
next person would have had to repeat.
A status row now leads: Ceph health, OSDs up and in, Ceph capacity used, and the
fullest EPHEMERAL volume across the three nodes. Each expression was run against
the live TSDB before being written in.
EPHEMERAL fill level gets its own graph too. It was the one number checked by
hand that had no home here, and it is the failure that has no safety net: Ceph
rebalances away from a full OSD, nothing rebalances away from a full /var — the
node simply stops.
The caveats panel gains three things found while answering the question:
Ceph does hold wear for the OSDs, in its devicehealth module and not in
Prometheus. The panel names the command rather than the current value, because a
number written into a dashboard is stale the moment it is written. It also
records that the retired Crucial drives were purged from Ceph's device registry,
so their history is not recoverable either — verified by asking for it and
getting ENOENT.
Talos boots with nvme_core.io_timeout=4294967295, so NVMe commands never time
out. A dying drive stalls instead of returning errors, and the latency panel
goes flat rather than spiking. That inverts what a reader would otherwise
conclude from a quiet graph.
Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and
a systemd-journal from elsewhere. I found this by searching Loki for NVMe errors,
getting nothing, and checking whether the collector existed before believing the
zero; it did not. talosctl dmesg is the only source, it is a ring buffer, and
this morning's reboots emptied it.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Throughput, IOPS, mean operation latency, utilisation, queue depth and
controller temperature for all nine NVMe drives, plus Ceph's own OSD latency for
comparison against the block layer.
Every panel joins node_disk_* to node_nvme_info and node_uname_info rather than
plotting the device name. That is not decoration: nvme0 is the Kingston system
disk on k8s-1 and k8s-3 and the Crucial on k8s-2, so a legend keyed on the
device compares different hardware across nodes and does so silently. The
inventory table resolves the OSD column through ceph_disk_occupation for the
same reason — it follows Rook instead of restating a device list that goes stale
the next time a drive is swapped.
Every expression was run against Prometheus before being written into the JSON,
which is also how the caveats in the panel descriptions were found. Latency is
accumulated-time over completed-operations, so it is a mean with no tail and it
vanishes while a device is idle: absence of traffic, not absence of a problem.
Utilisation near 100% on NVMe does not mean saturation the way it does on a
spindle, which is why queue depth shares that panel.
Temperature is the only SMART-adjacent value present, because it is the only one
the kernel publishes through sysfs. Wear, written-bytes totals, power-on hours
and media errors need an NVMe admin ioctl and no exporter here produces them
(#549). The dashboard says so in a panel rather than letting a green screen imply
a healthy drive.
That gap is worse than #549 describes, and the note records why. The Crucial P3
drives were retired from Ceph because they were worn, and now carry the
EPHEMERAL volume — so the drives with the least endurance left are the ones
holding container state, and they are the ones nothing watches. The measured
write latency already separates them from the Samsung OSDs by more than an order
of magnitude.
Also fixes the add-a-dashboard instructions above the generator, whose third
step told the reader to tick a row off ../DASHBOARD-MIGRATION.md. That file does
not exist anywhere in the repo, so the step could not be followed.
Refs #549
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No Proxmox remains. jupiter, saturn and neptune all run Talos on bare metal,
so everything that scraped, proxied, alerted on or rotated credentials for the
PVE API now points at hosts that answer nothing on :8006.
Two of these were not merely dead but actively failing on a schedule:
gatus an Infrastructure check on https://pve.renner.uno, every
60s, which can now only ever fail
openbao rotation a Class-B CronJob rolling the pve-exporter@pve API token
against pve.apps.svc.cluster.local:8006
Removed:
apps/pve/ Service, EndpointSlice (10.2.40.201-203),
IngressRoute for the PVE web UIs
monitoring/pve-exporter-* Deployment and its config ExternalSecret
prometheus scrape configs job_name pve and pve-node
dashboards/proxmox-cluster.json and its ConfigMap generator entry
image-automation/pve-exporter image policy for an image nothing runs
openbao/rotation/pve-exporter the rotation CronJob
openbao/policies/rotator-proxmox.hcl
crowdsec pve-ui-polling-whitelist, which suppressed false bans on PVE UI
API polling that no longer happens
Kept, with comments corrected: the insecure-backend ServersTransport, still
used by apps/file for the NAS web UI.
The gatus check leaves a numbering gap at "Infrastructure 2" rather than
renumbering 3 and 4. The labels are deliberately opaque and renumbering would
relabel the surviving checks, discarding their history for cosmetics.
Not touched: the opentofu Proxmox provider, VM and LXC modules,
vm_152_k8s_3.tf, sync_state_from_pve.py, and the ansible PVE inventory. Those
carry state implications and want a deliberate pass rather than folding into a
manifest cleanup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prometheus held 298,236 active series. ~108,000 of them -- over a third -- were
apiserver/etcd `_bucket` histograms. Each fans out to one series per bucket x
verb x resource x scope, so a single metric was 23,810 series on its own. Every
Home Assistant metric combined is ~4,100 for comparison.
That cardinality is the dominant driver of Prometheus write volume (~11 GB/day of
TSDB + WAL, doubled again by Ceph replication) and none of it is observable value
on three nodes: per-bucket apiserver latency quantiles are for clusters with SLAs
and dozens of nodes.
Dropped, all verified to have ZERO rule dependencies by checking each metric
against the live alert and recording rules:
apiserver_request_duration_seconds_bucket rules=0 23,810 series
apiserver_request_body_size_bytes_bucket rules=0 23,200
etcd_request_duration_seconds_bucket rules=0 15,840
apiserver_watch_list_duration_seconds_bucket rules=0 11,538
apiserver_watch_cache_read_wait_seconds_bucket rules=0 8,232
apiserver_response_sizes_bucket rules=0 7,592
apiserver_watch_events_sizes_bucket rules=0 5,220
-------
95,432
KEPT deliberately: apiserver_request_sli_duration_seconds_bucket (12,144 series).
23 recording rules depend on it, including the apiserver availability SLO and the
KubeAPIErrorBudgetBurn burn-rate series. Dropping it would silently break that
alerting -- the manifest says so inline, so a later "tidy up" of the regex does
not remove it by accident.
Only `_bucket` series are dropped. The matching `_count` and `_sum` remain, so
request rates and mean latencies still work; what is lost is per-bucket
quantiles.
etcd_* is included even though kubeEtcd is disabled: those are the apiserver's
own etcd-client metrics on the apiserver endpoint, not a separate etcd scrape.
Expected: ~32% fewer active series, so roughly 11 -> 7 GB/day of Prometheus
writes, ~8 GB/day off the drives once replication is counted. Will verify against
prometheus_tsdb_head_series rather than assume.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
Follow-up to the trmnl renderer finding. An `emptyDir` without `medium: Memory`
is NODE STORAGE, which is easy to read as "ephemeral, therefore free" -- it is
ephemeral, but every byte still hits the node's disk.
Survey of the repo: 23 emptyDir declarations, only 2 RAM-backed. This converts
12 of the remaining ones.
HONEST SCOPE: this buys almost no write reduction. Ranking every pod by OS-disk
writes over 6h shows trmnl-renderer at 67 GB/day (fixed separately) and then
nothing emptyDir-driven -- the rest of the top of that list is Ceph mons
(hostPath, not emptyDir), kube-apiserver, and container logs from databases.
Every volume converted here measured 8.0K in the running pod:
/var/lib/code-server 8.0K
/mosquitto/auth 8.0K
/state (z2m guards) 8.0K
So the value is correctness and, for mosquitto, keeping credential material off
the node's filesystem: that volume holds the generated passwd/acl files.
Each converted volume gets an explicit sizeLimit (16Mi where none existed), so
tmpfs cannot silently consume a container's memory limit. All are orders of
magnitude above measured usage.
Behaviour is unchanged otherwise: an emptyDir persists for the pod's lifetime on
either medium, so the mosquitto shim's assumption that the passwd file survives
a container restart still holds.
DELIBERATELY NOT CONVERTED:
- kubernetes/talos/cilium.yaml and talos/patches/cilium-cni.yaml -- vendored
and generated from the chart; a hand edit would be lost on upgrade.
- trmnl/renderer.yaml -- converted separately, where the actual 34 GB/day was.
Cost: this rolls the affected pods once.
Refs #434
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There are zero ceph_* series in Prometheus, and there have been for all 79
days of the cluster's life:
count of ceph_ metrics in Prometheus: 0
rook-ceph active scrape targets: 0
Not a Rook misconfiguration — `monitoring.enabled: true` is set and both
ServiceMonitors exist. They are simply never selected:
Prometheus serviceMonitorSelector : matchLabels {release: kube-prometheus-stack}
rook-ceph-mgr / rook-ceph-exporter: labels {team: rook}
The Rook operator creates those ServiceMonitors at runtime with its own
hardcoded label, so no static manifest or Kustomize patch in this repo can
relabel them.
Cluster-wide this was dropping 12 of 30 ServiceMonitors:
rook-ceph/rook-ceph-mgr, rook-ceph/rook-ceph-exporter (team: rook)
authentik/ak-outpost-* x10 (goauthentik.io/...)
Consequences: the entire storage layer had no telemetry and no history — so
Ceph trend analysis is impossible for anything before this lands, and any Ceph
PrometheusRule added today would have silently had nothing to fire on.
This is the ServiceMonitor twin of the trap already documented for rules in
monitoring-rules/prometheusrule-velero.yaml ("without this label the rule is
created but never loaded into Prometheus").
ruleSelector is deliberately NOT loosened — this repo's PrometheusRules all
carry the release label on purpose, and keeping that selector strict stops a
stray third-party rule from firing here.
Blast radius: 12 new scrape targets. The 10 authentik outposts come along with
the 2 rook ones; they are legitimate targets, but they are a side effect of
this change rather than its goal.
Found while investigating the Ceph BLUESTORE_SLOW_OP_ALERT warning, which
itself turned out to be a benign self-limiting event.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The alert triage agents could reach Prometheus and Alertmanager through the
apiserver service proxy, but not Loki:
kubectl auth can-i get "services/loki:3100" --subresource=proxy -n monitoring \
--as=system:serviceaccount:monitoring:ci-diagnostics -> no
(same check for kube-prometheus-stack-prometheus:9090 -> yes)
That gap has a sharp edge. A KubeJobFailed-class alert can only be cleared by
DELETING the failed Job — the rule's own annotation instructs this — and the
deletion destroys the pod and its logs. So the agents would inspect a failed
Job, find no live pod, declare the root cause "unconfirmable", and then delete
the last copy of the evidence.
The evidence was never actually gone: Alloy ships all pod logs to Loki with
retention_period: 2160h (90d). Issue #380 declared velero/restore-test-29740440
"probably unconfirmable" and its full failure output was in Loki the whole time
(FAIL: 3 — apps restore phase + two PVC parity checks).
Adds `loki` / `loki:3100` to the existing service-proxy Role (read-only GET,
same bare + :port-qualified pattern the other entries use, verified against the
live Service which exposes http-metrics:3100), and teaches both agent prompts
the query recipe so "the pod is gone" stops being accepted as a dead end.
No new privilege class: the agents already hold pods/log cluster-wide. This
mainly extends their reach to *deleted* pods.
Refs #427, #380
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes the vantage-point gap that let the 2026-07-24 controller outage
run ~20 h unnoticed: every existing probe of the public names resolves
in-cluster (CoreDNS/k8s_gateway) or gets hairpinned, so nothing measured
the actual WAN path.
- blackbox module http_2xx_wan_edge: probes the edge by proxy IP
literal with SNI/Host pinned to home.renner.uno. The Server/Ingress
VLANs are deliberately absent from the UniFi hairpin, so IP-literal
probes from the in-cluster exporter genuinely leave via the WAN and
come back through the netcup edge (verified live from a pod: v4
returns 200; v6 currently fails against proxy-1 — real breakage,
fixed separately by the resolv.conf PR).
- New probes: WAN edge per member/family (proxy-2 v4, proxy-1 v6),
https://netbird.renner.uno (controller), relay :9000/health on both
netcup members, tcp :33080 relay listeners.
- New PrometheusRule netbird-availability: controller unreachable
(critical 10m), whole edge down from WAN (critical 5m), single edge
member down (warning 15m), relay member down (warning 15m).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step 2 of the InfluxDB seed->apps handoff. #388 already landed
prune: disabled on the live HelmRelease/HelmRepository/ExternalSecret, so
retiring seed-influxdb here cannot trigger a helm uninstall — GC skips the
annotated objects and the new influxdb KS adopts them in place.
- git mv kubernetes/seed/influxdb/ -> kubernetes/apps/influxdb/ (annotations
from #388 travel with the files).
- Add kubernetes/clusters/homelab/influxdb.yaml (KS 'influxdb', path
apps/influxdb, deps infrastructure/crds/ceph). Delete seed-influxdb.yaml.
Review fixes from the closed atomic PR (#387):
- bootstrap.sh: wait on 'influxdb' instead of the deleted 'seed-influxdb'
(would otherwise spin the full 600s timeout on cold bootstrap).
- e2e-test.sh: swap the seed-influxdb readiness check for 'influxdb'.
- Fix stale comment paths kubernetes/seed/influxdb/... -> apps/influxdb/... in
network-policies/home-assistant-ns.yaml and apps/monitoring/
helmrelease-prometheus.yaml.
- Trim the helmrelease.yaml comment that described the removed seed-job.yaml,
and the ha-token-job.yaml 'at cutover moves to apps/influxdb' note (this is
that cutover).
Left intentionally: externalsecret.yaml key 'seed/influxdb/bootstrap' is an
OpenBao KV path, not a repo path — moving it needs a vault-side secret
migration, out of scope here.
Verified: kubectl kustomize apps/influxdb builds all objects with the right
prune/force annotations; bash -n passes on bootstrap.sh + e2e-test.sh; no
remaining seed-influxdb references in the repo (only the OpenBao KV path).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
*.k.renner.uno was a temporary migration namespace and is banned (owner
directive 2026-07-14, documented in openbao/ingress.yaml) — it was never
the live naming. Removes the .k. hosts/SANs from ical-proxy, loki,
grafana and the mosquitto cert, renames syslog.k.renner.uno to
syslog.renner.uno (cert + external-dns annotation; any syslog sender
still configured with the k. name must be repointed), fixes the stale
comments, updates e2e-test.sh probes to the real names, and drops
k.renner.uno from the external-dns domainFilters so leftover k. records
get garbage-collected by policy: sync.
The migration script's historical docstring reference stays — it
documents what the migration did.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
claude-botandClaude Opus 5.5
k.renner
renovate-bot
issue-fix-bot
security-bot