Compare commits

...
Author SHA1 Message Date
claude-bot da1482e33f docs: cut the essays out of the manifests I wrote today
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Config files were carrying investigation write-ups: two ruled-out hypotheses in
the smartctl-exporter securityContext, a twenty-line canonical note on the
Hetzner credential, the history of an alert threshold. That is derivation, not
configuration, and it belongs where it does not rot — the commit message for how
we got here, the wiki for what holds.

The distinction this applies: a comment stays if it changes what someone editing
that line would do — a constraint of the tool, a trap, a pointer to an issue. It
goes if it narrates how a conclusion was reached, restates a measurement, or
justifies a value after the fact. The test is whether the surrounding config
could change and leave the comment quietly wrong.

160 lines out, 30 in. Nothing behavioural changed; both manifests still pass a
server-side dry run.

Alert descriptions were tightened rather than removed. Those are not comments —
they are what someone reads when paged, and an essay is worse there than
anywhere.

This covers only what I added today. The repo has roughly 8600 comment lines
with 88 files over 25, and going further needs the criterion above agreed
first — much of that volume is load-bearing warning rather than narration.
2026-08-27 21:37:20 +02:00
k.renner c6b1aa109b Merge pull request 'refactor(secrets): drop the unused barman-cloud-creds' (#677) from drop-unused-barman-creds into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#677
2026-08-27 19:20:01 +00:00
claude-bot 2f3d952656 refactor(secrets): drop the unused barman-cloud-creds
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Three ExternalSecrets synced a Secret that nothing reads. Established rather
than assumed, from three directions: every live ObjectStore names
hetzner-s3-creds, no CNPG Cluster uses the legacy inline barmanObjectStore
(all seven run the barman-cloud plugin against an ObjectStore CR), and no pod
in the cluster mounts barman-cloud-creds.

They are almost certainly left from before the plugin migration, when the
inline configuration read a Secret of that name.

Worth stating plainly because it is not merely untidy: seed/{authentik,immich,
paperless}/barman-creds hold a Hetzner S3 credential that is NOT the one in
use. Fingerprint c2c5cf3e against 87954b64 for the live hetzner-s3 path, with
the three copies identical to each other. A valid key with no consumer is
exposure without benefit, so if it is a distinct S3 key it wants revoking at
Hetzner, which only a human can do — deleting our copies does not.

Removing the manifests does NOT remove the objects. All three carry
kustomize.toolkit.fluxcd.io/prune: disabled from the seed->app-KS handoffs, so
Flux skips them during garbage collection by design. The ExternalSecrets, their
Secrets and the three OpenBao paths have to be deleted explicitly after this
merges; git alone leaves them live and orphaned, which is worse than leaving
them declared.

The OpenBao paths deliberately stay until then. Deleting them first would leave
three ExternalSecrets failing to sync against a key that no longer exists, for
no gain.
2026-08-27 21:18:08 +02:00
k.renner 0a87f91116 Merge pull request 'refactor(secrets): one OpenBao path for the one Hetzner S3 credential' (#676) from hetzner-creds-shared-path into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#676
2026-08-27 19:04:23 +00:00
claude-bot c59f0050b3 refactor(secrets): one OpenBao path for the one Hetzner S3 credential
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#397 asked for the opposite: give woodpecker its own path so a rotation of
paperless's credentials would stop silently changing woodpecker's. The premise
does not hold. There is one Hetzner S3 key, and rotating it at Hetzner changes
it for every consumer no matter how many paths hold a copy. A fifth copy would
have bought nothing and made the separation look real.

Seven ExternalSecrets were reading it from four paths — seed/paperless (four
namespaces), seed/authentik, seed/immich, seed/trmnl. They now all read
infrastructure/hetzner-s3, which sits beside the existing
infrastructure/velero-hetzner.

That the four paths are byte-identical was verified rather than assumed, and
without reading them: SHA-256 over the three fields of each synced Secret
returns the same fingerprint in all seven namespaces. The same check says
infrastructure/velero-hetzner carries the same access key id, in a different
shape — a single `cloud` field holding an AWS credentials file.

Velero is deliberately not folded in. Moving it means templating that INI from
these fields, which is worth doing and not worth bundling with a path swap that
touches every database backup at once.

The reasoning lives once, in the paperless manifest, with the others pointing at
it. Repeating it six times is how six copies drift.

Kept from the note this replaces, because it was right and cost someone an
afternoon: the 403 recorded in #397 was a wrong-path error, not missing
permissions. The mount is `homelab`, so the path is homelab/seed/... —
`bao kv list homelab/seed` works with the identity available here.

homelab/infrastructure/hetzner-s3 now exists, copied from the paperless path
inside OpenBao so the value never left it, and fingerprints identical to its
source. Writes turned out to be possible all along: they fail through
openbao.renner.uno with a response the client cannot parse, and succeed against
the Service. That is worth knowing before the next person concludes, as I first
did, that the identity is read-only.

Closes #397
2026-08-27 21:02:28 +02:00
k.renner 48c46610a6 Merge pull request 'docs(monitoring): stop treating a worn EPHEMERAL drive as a hazard' (#674) from ephemeral-framing into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#674
2026-08-27 18:38:33 +00:00
claude-botandClaude Opus 5 53887dd88f docs(monitoring): stop treating a worn EPHEMERAL drive as a hazard
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
I described the worn drives carrying EPHEMERAL as the dangerous placement,
repeatedly, in the alert text and in the dashboard's caveats. That reads the
volume backwards. EPHEMERAL is Talos's /var and its contents are meant to be
disposable — container images, kubelet state, pod scratch, spegel and Rook's
local state all rebuild. Putting the drives with the least endurance left there
is a sensible destination, not a risk someone should be warned about.

The alert said "a node simply stops when it goes", which is simply wrong. A node
losing /var rebuilds it.

What survives the correction is narrower and checked rather than recalled:
/var/lib/etcd is on that volume too, so losing EPHEMERAL on a control-plane node
costs that member's data. Quorum holds at two of three and the member is removed
and re-added. That is a rebuild, not data loss, and the alert now says so at
that weight.

The fill-level panel keeps its warning but for the right reason: a full /var
stops a node being useful long before anything is lost, and nothing rebalances
it the way Ceph rebalances a full OSD. That is a capacity argument, not an
endurance one.

None of this changes the monitoring — measuring every drive is still right, and
the wear figures are unchanged. It changes what a reader is told the numbers
mean, which is the part that was misleading.

Refs #549

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-27 20:35:11 +02:00
k.renner c9a10ab015 Merge pull request 'feat(monitoring): put the SMART values on the dashboard' (#672) from disk-dashboard-smart into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#672
2026-08-27 13:27:36 +00:00
17 changed files with 48 additions and 219 deletions
@@ -1,18 +0,0 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: barman-cloud-creds
namespace: authentik
annotations:
# Prune-disabled — a standing guard kept from the seed->authentik-KS handoff (barman creds).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
dataFrom:
- extract:
key: seed/authentik/barman-creds
@@ -15,4 +15,4 @@ spec:
name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
dataFrom:
- extract:
key: seed/authentik/hetzner-creds
key: infrastructure/hetzner-s3
@@ -27,5 +27,4 @@ resources:
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- barman-creds-externalsecret.yaml
- hetzner-creds-externalsecret.yaml
@@ -1,21 +1,8 @@
---
# Barman S3 credentials for ha-pg.
#
# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
# problem and the same note. The values are the same today, so this is correct
# but coupled: rotating paperless's Hetzner credentials would silently change
# the backup credentials for Home Assistant and Woodpecker too.
#
# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
#
# bao kv put homelab/seed/home-assistant/hetzner-creds \
# @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
#
# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
# 403 on `bao kv list seed`, which was a wrong-path error rather than a
# permissions problem.
# One shared path for one shared credential — see the note in
# ../paperless/hetzner-creds-externalsecret.yaml.
#
# The namespace allowlist on the openbao ClusterSecretStore
# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -35,4 +22,4 @@ spec:
name: hetzner-s3-creds
dataFrom:
- extract:
key: seed/paperless/hetzner-creds
key: infrastructure/hetzner-s3
@@ -1,18 +0,0 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: barman-cloud-creds
namespace: immich
annotations:
# Prune-disabled — a standing guard kept from the seed->immich-KS handoff (barman creds).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
dataFrom:
- extract:
key: seed/immich/barman-creds
@@ -15,4 +15,4 @@ spec:
name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
dataFrom:
- extract:
key: seed/immich/hetzner-creds
key: infrastructure/hetzner-s3
@@ -13,6 +13,5 @@ resources:
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- barman-creds-externalsecret.yaml
- hetzner-creds-externalsecret.yaml
- pvcs.yaml
@@ -11,25 +11,10 @@ spec:
groups:
- name: disk-wear.rules
rules:
# THE GAP THESE RULES CLOSE
#
# Drive wear is invisible in every other signal this cluster has.
# `ceph -s` reads active+clean at any wear level, no Ceph health check
# fires until a device actually fails, and the block-layer metrics show
# throughput and latency but never endurance. That blind spot is how a
# set of OSD drives reached 81-87% before anyone looked (#549, #434).
#
# The numbers below are wear percentages reported by the drive itself:
# 100 means the manufacturer's rated endurance is used up. Drives do
# not stop at 100 — they typically keep working and start consuming
# spare blocks — so treat these as planning thresholds, not failure
# predictions.
# Wear percentages are self-reported: 100 means the rated endurance is
# spent, not that the drive stops. Planning thresholds, not predictions.
- alert: NvmeDriveWearHigh
# 80% with months of runway left is the point where ordering a
# replacement is still a calm decision rather than an outage.
# `for` is long because wear is monotonic and slow: it cannot
# flap, and a short window would only make the alert noisier
# after a restart.
# Long `for` because wear is monotonic and cannot flap.
expr: smartctl_device_percentage_used >= 80
for: 1h
labels:
@@ -37,12 +22,11 @@ spec:
annotations:
summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
description: >-
The drive has used {{ $value }}% of its rated write endurance.
Nothing is failing and no data is at risk yet. Check what the
drive is carrying before deciding urgency: an OSD is replaceable
with a rebalance, whereas the EPHEMERAL volume has no such
escape — a node simply stops when it goes. The Disk Health
dashboard resolves which drive holds which role.
{{ $value }}% of rated write endurance used. Nothing is failing
yet. Urgency depends on what the drive carries — the Disk Health
dashboard shows the role. Losing an OSD costs a rebalance;
EPHEMERAL is /var and rebuilds, apart from the node's etcd member
data, which has to be removed and re-added.
- alert: NvmeDriveWearCritical
expr: smartctl_device_percentage_used >= 95
for: 1h
@@ -55,9 +39,7 @@ spec:
likely keep working for a while, but it is now operating outside
what the manufacturer stands behind, and replacement should be
scheduled rather than considered.
# A drive that has consumed its spare area is no longer masking media
# defects, which is a materially different state from being worn:
# wear is a budget, spare exhaustion is damage.
# Wear is a budget; spare exhaustion is damage.
- alert: NvmeDriveSpareExhausted
expr: smartctl_device_available_spare <= smartctl_device_available_spare_threshold
for: 15m
@@ -70,10 +52,6 @@ spec:
drive has run out of reserve blocks to remap failures into.
Replace it.
- alert: NvmeDriveCriticalWarning
# The NVMe critical warning byte is a bitfield: spare below
# threshold, temperature out of range, degraded reliability,
# read-only mode, volatile-memory backup failed. Any non-zero value
# is the drive itself asking for attention.
expr: smartctl_device_critical_warning > 0
for: 5m
labels:
@@ -85,10 +63,8 @@ spec:
detail with `smartctl -a` on the node; the value is a bitfield,
not a count.
- alert: NvmeDriveMediaErrors
# Unlike wear, this counter should never move. Alert on the increase
# rather than the absolute value, so a drive that recorded errors
# long ago does not alert forever while a drive that starts failing
# today does.
# On the increase, not the total: pre-existing errors would otherwise
# alert forever. Standing counts are visible on the dashboard only.
expr: increase(smartctl_device_media_errors[6h]) > 0
for: 5m
labels:
@@ -99,14 +75,9 @@ spec:
The drive reported unrecoverable data-integrity errors in the
last six hours. A healthy drive never increments this. Check
whether Ceph has logged scrub errors on the same node.
# Without this the whole group is silently useless: no series means no
# alert fires, and a dashboard of empty panels reads much like a
# healthy one. This is the check that the checks are running.
# The check that the checks are running: no series fires no alert.
# Counted against node-exporter's inventory, not a fixed fleet size.
- alert: SmartctlExporterIncomplete
# Compared against node-exporter's own NVMe inventory rather than a
# written-down fleet size. Adding or removing a drive then moves both
# sides together, so this cannot rot into a permanent false alarm the
# way a hardcoded count would.
expr: count(smartctl_device_percentage_used) < count(node_nvme_info) or absent(smartctl_device_percentage_used)
for: 30m
labels:
@@ -14,16 +14,12 @@ spec:
matchLabels:
app.kubernetes.io/name: smartctl-exporter
endpoints:
# smartctl itself polls the drives every 60s, so scraping faster than that
# returns the same numbers with a new timestamp. Wear moves on a scale of
# weeks; 60s is already far finer than the signal.
# smartctl polls the drives every 60s; scraping faster gains nothing.
- port: http-metrics
path: /metrics
interval: 60s
relabelings:
# Without this every series is labelled with the exporter pod's IP and
# nothing else. A DaemonSet pod's IP says nothing about which machine
# the drive is in, and it changes on every reschedule — so the wear
# figure would be unattributable, which is most of its value gone.
# Without this a series carries only the exporter pod's IP, which does
# not identify the machine the drive is in.
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: node
@@ -70,7 +70,7 @@
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Fullest EPHEMERAL volume across the fleet — Talos's /var, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.",
"description": "Fullest EPHEMERAL volume across the fleet — Talos's /var. Its contents are meant to be disposable: container images, kubelet state and pod scratch all rebuild. Filling it is still a problem, because a node with no room to write stops being useful long before anything is lost, and nothing rebalances /var the way Ceph rebalances a full OSD.",
"fieldConfig": {
"defaults": {
"color": {"mode": "thresholds"},
@@ -453,7 +453,7 @@
"id": 8,
"options": {
"code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
"content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
"content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, and putting a worn one on EPHEMERAL is a sensible destination rather than a hazard — that volume is built to be rebuilt. Check the inventory table for what a given drive holds before reading its wear as urgent.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
"mode": "markdown"
},
"title": "Gaps and caveats",
@@ -1,37 +1,8 @@
# SMART/endurance metrics for every NVMe drive in the fleet (#549).
#
# WHY THIS HOLDS ELEVATED CAPABILITIES, and why that is a deliberate exception
# rather than an oversight:
#
# NVMe reports health over an admin ioctl on the character device
# (NVME_IOCTL_ADMIN_CMD), which the kernel gates on CAP_SYS_ADMIN. There is no
# sysfs path for wear, media errors or written-bytes totals — hwmon publishes
# the controller temperature and nothing else. Upstream smartctl_exporter
# documents `privileged: true` and `user: root` for exactly this reason.
#
# The alternative that needs no privileges — reading Ceph's devicehealth
# archive, which does hold full smartctl output — covers OSD drives only. In
# this cluster that is three of nine, and the six it misses include the drives
# that were retired from Ceph for wear and now carry EPHEMERAL. Covering the
# healthy drives and not the worn ones was judged not worth a new service; the
# owner chose fleet-wide coverage with the privilege exception instead.
#
# The exception is `privileged: true`, and narrower settings were tried and
# measured to fail — see the securityContext below for what was ruled out and
# how. It is still bounded: no host namespaces, no hostPID, no hostNetwork,
# read-only root filesystem, and /dev as the only host mount.
#
# If this ever needs revisiting, there are exactly two things worth checking,
# and neither is "does another exporter ask for less":
# - whether the kernel has grown a sysfs interface for NVMe health, which
# would remove the need for device access altogether
# - whether enough workloads need raw device access to justify a device
# plugin, which is the only way Kubernetes grants individual devices
# The requirement comes from the device cgroup and the ioctl gate, not from
# the choice of tool.
#
# Upstream had no release between 2025-04-22 (v0.14.0) and this deployment.
# The repository is not archived; the pin is simply its newest tag.
# Runs privileged: host device access is gated by the container device cgroup,
# and Kubernetes offers no per-device allowlist. Narrower settings do not work;
# the alternatives that were tried are in the commit history.
apiVersion: apps/v1
kind: DaemonSet
metadata:
@@ -57,10 +28,8 @@ spec:
containers:
- name: smartctl-exporter
image: quay.io/prometheuscommunity/smartctl-exporter:v0.14.0
# No --smartctl.device on purpose: the exporter scans for devices
# itself and rescans every 10m by default. Device names are not
# stable across these machines, so a hardcoded list would quietly
# stop covering a drive the next time one is replaced.
# No --smartctl.device: the exporter scans and rescans on its own.
# Device names are not stable across these machines.
args:
- --web.listen-address=:9633
ports:
@@ -68,56 +37,18 @@ spec:
containerPort: 9633
protocol: TCP
securityContext:
# PRIVILEGED, and there is no narrower setting that works. This
# was established by measurement after two narrower attempts
# failed, so the reasoning is recorded here rather than rediscovered:
#
# Attempt 1 — CAP_SYS_ADMIN + CAP_SYS_RAWIO, everything else
# dropped. The exporter found all three drives per node and
# could open none: "Smartctl open device: /dev/nvme0 failed:
# Operation not permitted".
#
# Attempt 2 — the same plus seccompProfile: Unconfined, on the
# theory that EPERM is the default seccomp action for an
# unlisted syscall and the kubelet runs seccompDefault: true.
# /proc/1/status showed CapEff 0x220000 and Seccomp 2, so both
# capabilities were effective and a filter was loaded. Removing
# the filter changed nothing.
#
# What actually settled it: `head -c 512 /dev/nvme0n1` inside
# the container fails with EPERM too. A plain read of a block
# device involves no ioctl, no capability and no syscall a
# seccomp profile would filter. The block is the device cgroup —
# containerd denies non-privileged containers access to host
# device nodes whether or not /dev is mounted.
#
# Kubernetes exposes no per-device allowlist in a Pod spec; there
# is no equivalent of `docker --device`. The alternatives are
# `privileged: true` or a device plugin advertising the NVMe
# drives as a resource. A device plugin is the genuinely narrower
# answer and is worth building if more workloads ever need raw
# device access; for a single read-only metrics exporter it is a
# component to write, deploy and maintain in exchange for the same
# access.
runAsUser: 0
runAsNonRoot: false
privileged: true
# Stated rather than omitted: the API rejects `privileged: true`
# together with `allowPrivilegeEscalation: false`, so leaving it
# out would only hide that privileged already implies it.
# The API rejects privileged together with
# allowPrivilegeEscalation: false.
allowPrivilegeEscalation: true
readOnlyRootFilesystem: true
volumeMounts:
# Not read-only. smartmontools opens NVMe devices read-write on
# some paths, and once the container is privileged a read-only
# flag on this mount is not a boundary anyone should rely on —
# keeping it would suggest a protection that is not there.
# Not read-only: smartmontools opens NVMe devices read-write.
- name: dev
mountPath: /dev
# smartctl is not known to need scratch space, but the root
# filesystem is read-only and a tool that turns out to want /tmp
# would crashloop instead of degrading. Cheaper to provide it than
# to find out from a restart counter.
# Scratch space, because the root filesystem is read-only.
- name: tmp
mountPath: /tmp
resources:
@@ -1,18 +0,0 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: barman-cloud-creds
namespace: paperless
annotations:
# Prune-disabled — a standing guard kept from the seed->paperless-KS handoff (barman creds).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
dataFrom:
- extract:
key: seed/paperless/barman-creds
@@ -1,3 +1,9 @@
# One Hetzner S3 key, one path, shared by every barman ObjectStore. Separate
# per-app paths would not make rotation independent — that needs one Hetzner
# key per consumer (#397).
#
# velero reads the same key from infrastructure/velero-hetzner, which holds it
# as a credentials file rather than these three fields.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
@@ -15,4 +21,4 @@ spec:
name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
dataFrom:
- extract:
key: seed/paperless/hetzner-creds
key: infrastructure/hetzner-s3
@@ -20,7 +20,6 @@ resources:
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- barman-creds-externalsecret.yaml
- hetzner-creds-externalsecret.yaml
- pvcs.yaml
configMapGenerator:
@@ -12,4 +12,4 @@ spec:
name: hetzner-s3-creds
dataFrom:
- extract:
key: seed/trmnl/hetzner-creds
key: infrastructure/hetzner-s3
@@ -1,11 +1,8 @@
---
# Barman S3 credentials for wp-pg.
#
# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
# that path needs OpenBao write access (`bao kv list seed` → 403 for the
# identities available here), so it stays open. The values are identical, so it
# is a one-line swap of the `key:` below once the path exists.
# One shared path for one shared credential — see the note in
# ../paperless/hetzner-creds-externalsecret.yaml.
#
# The namespace allowlist on the openbao ClusterSecretStore
# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -24,4 +21,4 @@ spec:
name: hetzner-s3-creds
dataFrom:
- extract:
key: seed/paperless/hetzner-creds
key: infrastructure/hetzner-s3
@@ -36,11 +36,9 @@ spec:
target:
name: hetzner-s3-creds
dataFrom:
# Reuses the paperless copy of the shared Hetzner S3 credentials (identical
# ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
# path for this migration — same approach woodpecker took. FOLLOW-UP: give
# wordpress its own seed/wordpress/hetzner-creds path once OpenBao can be
# written, and repoint this here (the value is the same, so it's a drop-in
# swap).
# One shared path for one shared credential. There is a single Hetzner S3
# key behind all of these — see the note in
# ../paperless/hetzner-creds-externalsecret.yaml for why per-app paths were
# the wrong shape and what would have to change for them to be right.
- extract:
key: seed/paperless/hetzner-creds
key: infrastructure/hetzner-s3