diff --git a/kubernetes/apps/monitoring/smartctl-exporter.yaml b/kubernetes/apps/monitoring/smartctl-exporter.yaml index 172b61af..72e75a66 100644 --- a/kubernetes/apps/monitoring/smartctl-exporter.yaml +++ b/kubernetes/apps/monitoring/smartctl-exporter.yaml @@ -16,17 +16,19 @@ # healthy drives and not the worn ones was judged not worth a new service; the # owner chose fleet-wide coverage with the privilege exception instead. # -# The exception is kept as small as the kernel allows: no `privileged: true`, -# no host namespaces, no hostPID. Only CAP_SYS_ADMIN and CAP_SYS_RAWIO are -# added, everything else is dropped, and the only host mount is /dev — which -# is required because the exporter must open the NVMe character devices and -# Kubernetes has no way to pass individual character devices to a pod without -# a device plugin. +# The exception is `privileged: true`, and narrower settings were tried and +# measured to fail — see the securityContext below for what was ruled out and +# how. It is still bounded: no host namespaces, no hostPID, no hostNetwork, +# read-only root filesystem, and /dev as the only host mount. # -# If this ever needs revisiting: the thing to check is whether the kernel has -# grown a sysfs interface for NVMe health, not whether some other exporter -# needs fewer privileges. The requirement comes from the ioctl gate, not from -# the tool. +# If this ever needs revisiting, there are exactly two things worth checking, +# and neither is "does another exporter ask for less": +# - whether the kernel has grown a sysfs interface for NVMe health, which +# would remove the need for device access altogether +# - whether enough workloads need raw device access to justify a device +# plugin, which is the only way Kubernetes grants individual devices +# The requirement comes from the device cgroup and the ioctl gate, not from +# the choice of tool. # # Upstream had no release between 2025-04-22 (v0.14.0) and this deployment. # The repository is not archived; the pin is simply its newest tag. @@ -66,38 +68,52 @@ spec: containerPort: 9633 protocol: TCP securityContext: - # Root plus two capabilities, NOT privileged. See the header. + # PRIVILEGED, and there is no narrower setting that works. This + # was established by measurement after two narrower attempts + # failed, so the reasoning is recorded here rather than rediscovered: + # + # Attempt 1 — CAP_SYS_ADMIN + CAP_SYS_RAWIO, everything else + # dropped. The exporter found all three drives per node and + # could open none: "Smartctl open device: /dev/nvme0 failed: + # Operation not permitted". + # + # Attempt 2 — the same plus seccompProfile: Unconfined, on the + # theory that EPERM is the default seccomp action for an + # unlisted syscall and the kubelet runs seccompDefault: true. + # /proc/1/status showed CapEff 0x220000 and Seccomp 2, so both + # capabilities were effective and a filter was loaded. Removing + # the filter changed nothing. + # + # What actually settled it: `head -c 512 /dev/nvme0n1` inside + # the container fails with EPERM too. A plain read of a block + # device involves no ioctl, no capability and no syscall a + # seccomp profile would filter. The block is the device cgroup — + # containerd denies non-privileged containers access to host + # device nodes whether or not /dev is mounted. + # + # Kubernetes exposes no per-device allowlist in a Pod spec; there + # is no equivalent of `docker --device`. The alternatives are + # `privileged: true` or a device plugin advertising the NVMe + # drives as a resource. A device plugin is the genuinely narrower + # answer and is worth building if more workloads ever need raw + # device access; for a single read-only metrics exporter it is a + # component to write, deploy and maintain in exchange for the same + # access. runAsUser: 0 runAsNonRoot: false - privileged: false - allowPrivilegeEscalation: false + privileged: true + # Stated rather than omitted: the API rejects `privileged: true` + # together with `allowPrivilegeEscalation: false`, so leaving it + # out would only hide that privileged already implies it. + allowPrivilegeEscalation: true readOnlyRootFilesystem: true - capabilities: - drop: ["ALL"] - add: ["SYS_ADMIN", "SYS_RAWIO"] - # The capabilities alone are not enough, and this was measured - # rather than assumed. With the default profile the exporter found - # all three drives and could open none of them: - # "Smartctl open device: /dev/nvme0 failed: Operation not - # permitted" - # EPERM is the default seccomp action for a syscall the profile - # does not list. Reading /proc/1/status in the running container - # separated the two candidates: CapEff was 0x220000, so SYS_ADMIN - # and SYS_RAWIO were both effective — while Seccomp was 2 with one - # filter loaded. The kubelet sets seccompDefault: true, so every - # pod that does not say otherwise gets RuntimeDefault. - # - # This is still narrower than `privileged: true`, which would also - # grant every capability, all devices and write access to /sys. - # Narrowing further would mean shipping a custom seccomp profile - # that allows exactly the ioctl smartctl needs — worth doing if - # this list of exceptions ever grows, not worth it for one pod. - seccompProfile: - type: Unconfined volumeMounts: + # Not read-only. smartmontools opens NVMe devices read-write on + # some paths, and once the container is privileged a read-only + # flag on this mount is not a boundary anyone should rely on — + # keeping it would suggest a protection that is not there. - name: dev mountPath: /dev - readOnly: true # smartctl is not known to need scratch space, but the root # filesystem is read-only and a tool that turns out to want /tmp # would crashloop instead of degrading. Cheaper to provide it than