From d0bea0dc15c816a1245573a435c2b0dec2734d4b Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Thu, 27 Aug 2026 15:14:32 +0200 Subject: [PATCH] fix(monitoring): privileged is the mechanism, not the shortcut MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third attempt, and the first two were wrong for the same reason: I kept looking for the gate in the capability and syscall layers, and it was never there. Attempt one gave the container CAP_SYS_ADMIN and CAP_SYS_RAWIO with everything else dropped. It found all three drives per node and could open none of them. Attempt two added seccompProfile: Unconfined, reasoning that EPERM is the default action for a syscall an unlisted profile blocks and that the kubelet runs seccompDefault: true. /proc/1/status confirmed both capabilities effective and a filter loaded, so the theory was at least testable — and removing the filter changed nothing. What settled it was a smaller test than either: `head -c 512 /dev/nvme0n1` inside the container also fails with EPERM. Reading a block device involves no ioctl, no capability and nothing a seccomp profile filters. The denial is the device cgroup, which containerd applies to every non-privileged container regardless of whether /dev is mounted. Mounting /dev makes the nodes visible; it does not make them usable. Kubernetes has no per-device allowlist in a Pod spec — no equivalent of `docker --device`. So the honest position is that `privileged: true` here is not a lazy shortcut standing in for a narrower setting; it is the only mechanism the API offers, short of writing a device plugin. The manifest now says that, and records what was ruled out and how, so the next reader does not spend the same two rounds in the capability layer. The /dev mount loses its readOnly flag with it. smartmontools opens NVMe devices read-write on some paths, and once the container is privileged that flag suggests a boundary that is not there. A device plugin advertising the drives as a schedulable resource is the genuinely narrower answer and is noted as such. It is worth building when more than one workload needs raw device access; for a single read-only metrics exporter it is a component to write, deploy and maintain in exchange for the same access. Refs #549 Co-Authored-By: Claude Opus 5 (1M context) --- .../apps/monitoring/smartctl-exporter.yaml | 88 +++++++++++-------- 1 file changed, 52 insertions(+), 36 deletions(-) diff --git a/kubernetes/apps/monitoring/smartctl-exporter.yaml b/kubernetes/apps/monitoring/smartctl-exporter.yaml index 172b61af..72e75a66 100644 --- a/kubernetes/apps/monitoring/smartctl-exporter.yaml +++ b/kubernetes/apps/monitoring/smartctl-exporter.yaml @@ -16,17 +16,19 @@ # healthy drives and not the worn ones was judged not worth a new service; the # owner chose fleet-wide coverage with the privilege exception instead. # -# The exception is kept as small as the kernel allows: no `privileged: true`, -# no host namespaces, no hostPID. Only CAP_SYS_ADMIN and CAP_SYS_RAWIO are -# added, everything else is dropped, and the only host mount is /dev — which -# is required because the exporter must open the NVMe character devices and -# Kubernetes has no way to pass individual character devices to a pod without -# a device plugin. +# The exception is `privileged: true`, and narrower settings were tried and +# measured to fail — see the securityContext below for what was ruled out and +# how. It is still bounded: no host namespaces, no hostPID, no hostNetwork, +# read-only root filesystem, and /dev as the only host mount. # -# If this ever needs revisiting: the thing to check is whether the kernel has -# grown a sysfs interface for NVMe health, not whether some other exporter -# needs fewer privileges. The requirement comes from the ioctl gate, not from -# the tool. +# If this ever needs revisiting, there are exactly two things worth checking, +# and neither is "does another exporter ask for less": +# - whether the kernel has grown a sysfs interface for NVMe health, which +# would remove the need for device access altogether +# - whether enough workloads need raw device access to justify a device +# plugin, which is the only way Kubernetes grants individual devices +# The requirement comes from the device cgroup and the ioctl gate, not from +# the choice of tool. # # Upstream had no release between 2025-04-22 (v0.14.0) and this deployment. # The repository is not archived; the pin is simply its newest tag. @@ -66,38 +68,52 @@ spec: containerPort: 9633 protocol: TCP securityContext: - # Root plus two capabilities, NOT privileged. See the header. + # PRIVILEGED, and there is no narrower setting that works. This + # was established by measurement after two narrower attempts + # failed, so the reasoning is recorded here rather than rediscovered: + # + # Attempt 1 — CAP_SYS_ADMIN + CAP_SYS_RAWIO, everything else + # dropped. The exporter found all three drives per node and + # could open none: "Smartctl open device: /dev/nvme0 failed: + # Operation not permitted". + # + # Attempt 2 — the same plus seccompProfile: Unconfined, on the + # theory that EPERM is the default seccomp action for an + # unlisted syscall and the kubelet runs seccompDefault: true. + # /proc/1/status showed CapEff 0x220000 and Seccomp 2, so both + # capabilities were effective and a filter was loaded. Removing + # the filter changed nothing. + # + # What actually settled it: `head -c 512 /dev/nvme0n1` inside + # the container fails with EPERM too. A plain read of a block + # device involves no ioctl, no capability and no syscall a + # seccomp profile would filter. The block is the device cgroup — + # containerd denies non-privileged containers access to host + # device nodes whether or not /dev is mounted. + # + # Kubernetes exposes no per-device allowlist in a Pod spec; there + # is no equivalent of `docker --device`. The alternatives are + # `privileged: true` or a device plugin advertising the NVMe + # drives as a resource. A device plugin is the genuinely narrower + # answer and is worth building if more workloads ever need raw + # device access; for a single read-only metrics exporter it is a + # component to write, deploy and maintain in exchange for the same + # access. runAsUser: 0 runAsNonRoot: false - privileged: false - allowPrivilegeEscalation: false + privileged: true + # Stated rather than omitted: the API rejects `privileged: true` + # together with `allowPrivilegeEscalation: false`, so leaving it + # out would only hide that privileged already implies it. + allowPrivilegeEscalation: true readOnlyRootFilesystem: true - capabilities: - drop: ["ALL"] - add: ["SYS_ADMIN", "SYS_RAWIO"] - # The capabilities alone are not enough, and this was measured - # rather than assumed. With the default profile the exporter found - # all three drives and could open none of them: - # "Smartctl open device: /dev/nvme0 failed: Operation not - # permitted" - # EPERM is the default seccomp action for a syscall the profile - # does not list. Reading /proc/1/status in the running container - # separated the two candidates: CapEff was 0x220000, so SYS_ADMIN - # and SYS_RAWIO were both effective — while Seccomp was 2 with one - # filter loaded. The kubelet sets seccompDefault: true, so every - # pod that does not say otherwise gets RuntimeDefault. - # - # This is still narrower than `privileged: true`, which would also - # grant every capability, all devices and write access to /sys. - # Narrowing further would mean shipping a custom seccomp profile - # that allows exactly the ioctl smartctl needs — worth doing if - # this list of exceptions ever grows, not worth it for one pod. - seccompProfile: - type: Unconfined volumeMounts: + # Not read-only. smartmontools opens NVMe devices read-write on + # some paths, and once the container is privileged a read-only + # flag on this mount is not a boundary anyone should rely on — + # keeping it would suggest a protection that is not there. - name: dev mountPath: /dev - readOnly: true # smartctl is not known to need scratch space, but the root # filesystem is read-only and a tool that turns out to want /tmp # would crashloop instead of degrading. Cheaper to provide it than -- 2.54.0