fix(monitoring): privileged is the mechanism, not the shortcut #671

2026-08-27T13:15:53Z

claude-bot commented

2026-08-27 13:15:53 +00:00

Dritter Anlauf, und die ersten beiden waren aus demselben Grund falsch: ich habe den Blocker in der Capability- und Syscall-Ebene gesucht, und dort war er nie.

Der entscheidende Test war kleiner als beide Hypothesen. Im Container schlägt schon das hier fehl:

head: /dev/nvme0n1: Operation not permitted

Ein einfaches Lesen eines Blockgeräts — kein ioctl, keine Capability, nichts, was ein Seccomp-Profil filtert. Die Sperre ist die Device-Cgroup, die containerd auf jeden nicht-privilegierten Container anwendet, unabhängig davon, ob /dev gemountet ist. Der Mount macht die Gerätenoten sichtbar, nicht benutzbar.

Zum Vergleich, was ich vorher gemessen und falsch gedeutet habe: CapEff 0x220000 (beide Capabilities wirksam), Seccomp 2 mit einem Filter — und nach dem Entfernen des Filters exakt derselbe Fehler.

Kubernetes hat keine Geräte-Allowlist im Pod-Spec, kein Äquivalent zu docker --device. Damit ist privileged: true hier keine bequeme Abkürzung anstelle einer engeren Einstellung, sondern der einzige Mechanismus, den die API anbietet — außer man schreibt ein Device-Plugin. Meine Formulierung im vorigen PR, die Ausnahme sei „so klein wie der Kernel es zulässt", war damit sachlich falsch.

Das steht jetzt im Manifest, samt dem, was ausgeschlossen wurde und wie — damit der nächste Leser nicht dieselben zwei Runden in der Capability-Ebene dreht.

Der readOnly-Flag am /dev-Mount fällt mit weg. smartmontools öffnet NVMe-Geräte auf manchen Pfaden schreibend, und in einem privilegierten Container suggeriert das Flag eine Grenze, die es nicht gibt.

Ein Device-Plugin, das die Laufwerke als planbare Ressource anbietet, wäre die wirklich engere Antwort — vermerkt als solche. Das lohnt sich, wenn mehr als eine Workload Rohzugriff braucht; für einen lesenden Metrik-Exporter ist es eine Komponente zum Schreiben, Ausrollen und Pflegen für denselben Zugriff.

Geprüft mit --server-side --dry-run unter Flux' Field-Manager, weil der normale kubectl apply-Dry-Run das alte allowPrivilegeEscalation: false aus dem Live-Objekt mitmergt und daran scheitert.

Refs #549

Dritter Anlauf, und die ersten beiden waren aus demselben Grund falsch: ich habe den Blocker in der Capability- und Syscall-Ebene gesucht, und dort war er nie. **Der entscheidende Test war kleiner als beide Hypothesen.** Im Container schlägt schon das hier fehl: ``` head: /dev/nvme0n1: Operation not permitted ``` Ein einfaches Lesen eines Blockgeräts — kein ioctl, keine Capability, nichts, was ein Seccomp-Profil filtert. Die Sperre ist die **Device-Cgroup**, die containerd auf jeden nicht-privilegierten Container anwendet, unabhängig davon, ob `/dev` gemountet ist. Der Mount macht die Gerätenoten sichtbar, nicht benutzbar. Zum Vergleich, was ich vorher gemessen und falsch gedeutet habe: `CapEff 0x220000` (beide Capabilities wirksam), `Seccomp 2` mit einem Filter — und nach dem Entfernen des Filters exakt derselbe Fehler. **Kubernetes hat keine Geräte-Allowlist im Pod-Spec**, kein Äquivalent zu `docker --device`. Damit ist `privileged: true` hier keine bequeme Abkürzung anstelle einer engeren Einstellung, sondern der einzige Mechanismus, den die API anbietet — außer man schreibt ein Device-Plugin. Meine Formulierung im vorigen PR, die Ausnahme sei „so klein wie der Kernel es zulässt", war damit sachlich falsch. Das steht jetzt im Manifest, samt dem, was ausgeschlossen wurde und wie — damit der nächste Leser nicht dieselben zwei Runden in der Capability-Ebene dreht. Der `readOnly`-Flag am `/dev`-Mount fällt mit weg. smartmontools öffnet NVMe-Geräte auf manchen Pfaden schreibend, und in einem privilegierten Container suggeriert das Flag eine Grenze, die es nicht gibt. Ein Device-Plugin, das die Laufwerke als planbare Ressource anbietet, wäre die wirklich engere Antwort — vermerkt als solche. Das lohnt sich, wenn mehr als eine Workload Rohzugriff braucht; für einen lesenden Metrik-Exporter ist es eine Komponente zum Schreiben, Ausrollen und Pflegen für denselben Zugriff. Geprüft mit `--server-side --dry-run` unter Flux' Field-Manager, weil der normale `kubectl apply`-Dry-Run das alte `allowPrivilegeEscalation: false` aus dem Live-Objekt mitmergt und daran scheitert. Refs #549

claude-bot added 1 commit 2026-08-27 13:15:53 +00:00

fix(monitoring): privileged is the mechanism, not the shortcut

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

d0bea0dc15

Third attempt, and the first two were wrong for the same reason: I kept looking
for the gate in the capability and syscall layers, and it was never there.

Attempt one gave the container CAP_SYS_ADMIN and CAP_SYS_RAWIO with everything
else dropped. It found all three drives per node and could open none of them.
Attempt two added seccompProfile: Unconfined, reasoning that EPERM is the
default action for a syscall an unlisted profile blocks and that the kubelet
runs seccompDefault: true. /proc/1/status confirmed both capabilities effective
and a filter loaded, so the theory was at least testable — and removing the
filter changed nothing.

What settled it was a smaller test than either: `head -c 512 /dev/nvme0n1`
inside the container also fails with EPERM. Reading a block device involves no
ioctl, no capability and nothing a seccomp profile filters. The denial is the
device cgroup, which containerd applies to every non-privileged container
regardless of whether /dev is mounted. Mounting /dev makes the nodes visible;
it does not make them usable.

Kubernetes has no per-device allowlist in a Pod spec — no equivalent of
`docker --device`. So the honest position is that `privileged: true` here is
not a lazy shortcut standing in for a narrower setting; it is the only
mechanism the API offers, short of writing a device plugin. The manifest now
says that, and records what was ruled out and how, so the next reader does not
spend the same two rounds in the capability layer.

The /dev mount loses its readOnly flag with it. smartmontools opens NVMe
devices read-write on some paths, and once the container is privileged that
flag suggests a boundary that is not there.

A device plugin advertising the drives as a schedulable resource is the
genuinely narrower answer and is noted as such. It is worth building when more
than one workload needs raw device access; for a single read-only metrics
exporter it is a component to write, deploy and maintain in exchange for the
same access.

Refs #549

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit b8061ea05d into master

2026-08-27 13:19:38 +00:00

k.renner referenced this issue from a commit

2026-08-27 13:19:39 +00:00

Merge pull request 'fix(monitoring): privileged is the mechanism, not the shortcut' (#671) from smartctl-privileged into master

review-bot reviewed

2026-08-27 13:19:47 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit d0bea0dc15c816a1245573a435c2b0dec2734d4b.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Single-file change to kubernetes/apps/monitoring/smartctl-exporter.yaml: replaces the capability-scoped securityContext (CAP_SYS_ADMIN + CAP_SYS_RAWIO, seccompProfile: Unconfined) with privileged: true, and drops readOnly: true on the /dev hostPath mount. The comment block documents two prior failed narrower attempts and the measurement (head -c 512 /dev/nvme0n1 → EPERM) that pinned the real blocker on the device cgroup rather than capabilities or seccomp. That diagnosis is correct: containerd's default (non-privileged) device cgroup denies raw block/char device access regardless of granted capabilities, and Kubernetes has no per-device allowlist in the Pod API (no docker --device equivalent) — the only ways around it are privileged: true or a device plugin, exactly as the comment states. No blocking issues found.

🟡 Suggestions

kubernetes/apps/monitoring/smartctl-exporter.yaml:109 (readOnlyRootFilesystem: true) sits next to a mount comment (:111-114) arguing that a readOnly flag "suggest[s] a boundary that isn't there" once the container is privileged. The same logic applies to readOnlyRootFilesystem: a privileged container holds CAP_SYS_ADMIN and can remount its own rootfs read-write, so this flag is a soft guard against an accidental write, not an actual boundary against a hostile process — consistent with the reasoning that was just used to justify removing the analogous flag on /dev. Not asking for a change (soft guards still have some value against bugs, just not attackers), but the comment doesn't acknowledge the tension, and a future reader may wonder why one "not a real boundary" flag was dropped and the other kept.

🟢 Nits

Worth noting for other reviewers (not a change request): kubernetes/infrastructure/namespaces/namespaces.yaml:17-20 already sets pod-security.kubernetes.io/enforce: privileged on the monitoring namespace (and this is cluster-wide, per the repeated blocks in that file), so this change doesn't need any new PSA exception and won't be blocked by admission — it only widens what this one pod already had access to request.

Tests

No automated test exists or is expected for a manifest-only change in this repo; .woodpecker/ci-manifests-test.yml's kustomize-build step will validate the file still builds as part of the monitoring kustomization, which is the applicable gate here. The PR body states the change was additionally checked with kubectl apply --server-side --dry-run=server under Flux's field-manager (explaining why a plain client-side dry-run was avoided — it would merge in the old live allowPrivilegeEscalation: false and fail validation). That's reasonable verification for this kind of change; I have no way to re-run it from this checkout.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `d0bea0dc15c816a1245573a435c2b0dec2734d4b`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Single-file change to `kubernetes/apps/monitoring/smartctl-exporter.yaml`: replaces the capability-scoped `securityContext` (`CAP_SYS_ADMIN` + `CAP_SYS_RAWIO`, `seccompProfile: Unconfined`) with `privileged: true`, and drops `readOnly: true` on the `/dev` hostPath mount. The comment block documents two prior failed narrower attempts and the measurement (`head -c 512 /dev/nvme0n1` → `EPERM`) that pinned the real blocker on the device cgroup rather than capabilities or seccomp. That diagnosis is correct: containerd's default (non-privileged) device cgroup denies raw block/char device access regardless of granted capabilities, and Kubernetes has no per-device allowlist in the Pod API (no `docker --device` equivalent) — the only ways around it are `privileged: true` or a device plugin, exactly as the comment states. No blocking issues found. ## 🟡 Suggestions - `kubernetes/apps/monitoring/smartctl-exporter.yaml:109` (`readOnlyRootFilesystem: true`) sits next to a mount comment (`:111-114`) arguing that a `readOnly` flag "suggest[s] a boundary that isn't there" once the container is privileged. The same logic applies to `readOnlyRootFilesystem`: a privileged container holds `CAP_SYS_ADMIN` and can remount its own rootfs read-write, so this flag is a soft guard against an accidental write, not an actual boundary against a hostile process — consistent with the reasoning that was just used to justify removing the analogous flag on `/dev`. Not asking for a change (soft guards still have some value against bugs, just not attackers), but the comment doesn't acknowledge the tension, and a future reader may wonder why one "not a real boundary" flag was dropped and the other kept. ## 🟢 Nits - Worth noting for other reviewers (not a change request): `kubernetes/infrastructure/namespaces/namespaces.yaml:17-20` already sets `pod-security.kubernetes.io/enforce: privileged` on the `monitoring` namespace (and this is cluster-wide, per the repeated blocks in that file), so this change doesn't need any new PSA exception and won't be blocked by admission — it only widens what this one pod already had access to request. ## Tests No automated test exists or is expected for a manifest-only change in this repo; `.woodpecker/ci-manifests-test.yml`'s `kustomize-build` step will validate the file still builds as part of the `monitoring` kustomization, which is the applicable gate here. The PR body states the change was additionally checked with `kubectl apply --server-side --dry-run=server` under Flux's field-manager (explaining why a plain client-side dry-run was avoided — it would merge in the old live `allowPrivilegeEscalation: false` and fail validation). That's reasonable verification for this kind of change; I have no way to re-run it from this checkout. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#671