fix(monitoring): privileged is the mechanism, not the shortcut

Third attempt, and the first two were wrong for the same reason: I kept looking
for the gate in the capability and syscall layers, and it was never there.

Attempt one gave the container CAP_SYS_ADMIN and CAP_SYS_RAWIO with everything
else dropped. It found all three drives per node and could open none of them.
Attempt two added seccompProfile: Unconfined, reasoning that EPERM is the
default action for a syscall an unlisted profile blocks and that the kubelet
runs seccompDefault: true. /proc/1/status confirmed both capabilities effective
and a filter loaded, so the theory was at least testable — and removing the
filter changed nothing.

What settled it was a smaller test than either: `head -c 512 /dev/nvme0n1`
inside the container also fails with EPERM. Reading a block device involves no
ioctl, no capability and nothing a seccomp profile filters. The denial is the
device cgroup, which containerd applies to every non-privileged container
regardless of whether /dev is mounted. Mounting /dev makes the nodes visible;
it does not make them usable.

Kubernetes has no per-device allowlist in a Pod spec — no equivalent of
`docker --device`. So the honest position is that `privileged: true` here is
not a lazy shortcut standing in for a narrower setting; it is the only
mechanism the API offers, short of writing a device plugin. The manifest now
says that, and records what was ruled out and how, so the next reader does not
spend the same two rounds in the capability layer.

The /dev mount loses its readOnly flag with it. smartmontools opens NVMe
devices read-write on some paths, and once the container is privileged that
flag suggests a boundary that is not there.

A device plugin advertising the drives as a schedulable resource is the
genuinely narrower answer and is noted as such. It is worth building when more
than one workload needs raw device access; for a single read-only metrics
exporter it is a component to write, deploy and maintain in exchange for the
same access.

Refs #549

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/monitoring/smartctl-exporter.yaml

@@ -16,17 +16,19 @@
 # healthy drives and not the worn ones was judged not worth a new service; the
 # owner chose fleet-wide coverage with the privilege exception instead.
 #
-# The exception is kept as small as the kernel allows: no `privileged: true`,
-# no host namespaces, no hostPID. Only CAP_SYS_ADMIN and CAP_SYS_RAWIO are
-# added, everything else is dropped, and the only host mount is /dev — which
-# is required because the exporter must open the NVMe character devices and
-# Kubernetes has no way to pass individual character devices to a pod without
-# a device plugin.
+# The exception is `privileged: true`, and narrower settings were tried and
+# measured to fail — see the securityContext below for what was ruled out and
+# how. It is still bounded: no host namespaces, no hostPID, no hostNetwork,
+# read-only root filesystem, and /dev as the only host mount.
 #
-# If this ever needs revisiting: the thing to check is whether the kernel has
-# grown a sysfs interface for NVMe health, not whether some other exporter
-# needs fewer privileges. The requirement comes from the ioctl gate, not from
-# the tool.
+# If this ever needs revisiting, there are exactly two things worth checking,
+# and neither is "does another exporter ask for less":
+#   - whether the kernel has grown a sysfs interface for NVMe health, which
+#     would remove the need for device access altogether
+#   - whether enough workloads need raw device access to justify a device
+#     plugin, which is the only way Kubernetes grants individual devices
+# The requirement comes from the device cgroup and the ioctl gate, not from
+# the choice of tool.
 #
 # Upstream had no release between 2025-04-22 (v0.14.0) and this deployment.
 # The repository is not archived; the pin is simply its newest tag.
@@ -66,38 +68,52 @@ spec:
               containerPort: 9633
               protocol: TCP
           securityContext:
-            # Root plus two capabilities, NOT privileged. See the header.
+            # PRIVILEGED, and there is no narrower setting that works. This
+            # was established by measurement after two narrower attempts
+            # failed, so the reasoning is recorded here rather than rediscovered:
+            #
+            #   Attempt 1 — CAP_SYS_ADMIN + CAP_SYS_RAWIO, everything else
+            #   dropped. The exporter found all three drives per node and
+            #   could open none: "Smartctl open device: /dev/nvme0 failed:
+            #   Operation not permitted".
+            #
+            #   Attempt 2 — the same plus seccompProfile: Unconfined, on the
+            #   theory that EPERM is the default seccomp action for an
+            #   unlisted syscall and the kubelet runs seccompDefault: true.
+            #   /proc/1/status showed CapEff 0x220000 and Seccomp 2, so both
+            #   capabilities were effective and a filter was loaded. Removing
+            #   the filter changed nothing.
+            #
+            #   What actually settled it: `head -c 512 /dev/nvme0n1` inside
+            #   the container fails with EPERM too. A plain read of a block
+            #   device involves no ioctl, no capability and no syscall a
+            #   seccomp profile would filter. The block is the device cgroup —
+            #   containerd denies non-privileged containers access to host
+            #   device nodes whether or not /dev is mounted.
+            #
+            # Kubernetes exposes no per-device allowlist in a Pod spec; there
+            # is no equivalent of `docker --device`. The alternatives are
+            # `privileged: true` or a device plugin advertising the NVMe
+            # drives as a resource. A device plugin is the genuinely narrower
+            # answer and is worth building if more workloads ever need raw
+            # device access; for a single read-only metrics exporter it is a
+            # component to write, deploy and maintain in exchange for the same
+            # access.
             runAsUser: 0
             runAsNonRoot: false
-            privileged: false
-            allowPrivilegeEscalation: false
+            privileged: true
+            # Stated rather than omitted: the API rejects `privileged: true`
+            # together with `allowPrivilegeEscalation: false`, so leaving it
+            # out would only hide that privileged already implies it.
+            allowPrivilegeEscalation: true
             readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-              add: ["SYS_ADMIN", "SYS_RAWIO"]
-            # The capabilities alone are not enough, and this was measured
-            # rather than assumed. With the default profile the exporter found
-            # all three drives and could open none of them:
-            #   "Smartctl open device: /dev/nvme0 failed: Operation not
-            #   permitted"
-            # EPERM is the default seccomp action for a syscall the profile
-            # does not list. Reading /proc/1/status in the running container
-            # separated the two candidates: CapEff was 0x220000, so SYS_ADMIN
-            # and SYS_RAWIO were both effective — while Seccomp was 2 with one
-            # filter loaded. The kubelet sets seccompDefault: true, so every
-            # pod that does not say otherwise gets RuntimeDefault.
-            #
-            # This is still narrower than `privileged: true`, which would also
-            # grant every capability, all devices and write access to /sys.
-            # Narrowing further would mean shipping a custom seccomp profile
-            # that allows exactly the ioctl smartctl needs — worth doing if
-            # this list of exceptions ever grows, not worth it for one pod.
-            seccompProfile:
-              type: Unconfined
           volumeMounts:
+            # Not read-only. smartmontools opens NVMe devices read-write on
+            # some paths, and once the container is privileged a read-only
+            # flag on this mount is not a boundary anyone should rely on —
+            # keeping it would suggest a protection that is not there.
             - name: dev
               mountPath: /dev
-              readOnly: true
             # smartctl is not known to need scratch space, but the root
             # filesystem is read-only and a tool that turns out to want /tmp
             # would crashloop instead of degrading. Cheaper to provide it than