refactor(secrets): one OpenBao path for the one Hetzner S3 credential

#397 asked for the opposite: give woodpecker its own path so a rotation of
paperless's credentials would stop silently changing woodpecker's. The premise
does not hold. There is one Hetzner S3 key, and rotating it at Hetzner changes
it for every consumer no matter how many paths hold a copy. A fifth copy would
have bought nothing and made the separation look real.

Seven ExternalSecrets were reading it from four paths — seed/paperless (four
namespaces), seed/authentik, seed/immich, seed/trmnl. They now all read
infrastructure/hetzner-s3, which sits beside the existing
infrastructure/velero-hetzner.

That the four paths are byte-identical was verified rather than assumed, and
without reading them: SHA-256 over the three fields of each synced Secret
returns the same fingerprint in all seven namespaces. The same check says
infrastructure/velero-hetzner carries the same access key id, in a different
shape — a single `cloud` field holding an AWS credentials file.

Velero is deliberately not folded in. Moving it means templating that INI from
these fields, which is worth doing and not worth bundling with a path swap that
touches every database backup at once.

The reasoning lives once, in the paperless manifest, with the others pointing at
it. Repeating it six times is how six copies drift.

Kept from the note this replaces, because it was right and cost someone an
afternoon: the 403 recorded in #397 was a wrong-path error, not missing
permissions. The mount is `homelab`, so the path is homelab/seed/... —
`bao kv list homelab/seed` works with the identity available here.

homelab/infrastructure/hetzner-s3 now exists, copied from the paperless path
inside OpenBao so the value never left it, and fingerprints identical to its
source. Writes turned out to be possible all along: they fail through
openbao.renner.uno with a response the client cannot parse, and succeed against
the Service. That is worth knowing before the next person concludes, as I first
did, that the identity is read-only.

Closes #397

kubernetes/apps/authentik/hetzner-creds-externalsecret.yaml

@@ -15,4 +15,4 @@ spec:
     name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
   dataFrom:
     - extract:
-        key: seed/authentik/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml

@@ -1,21 +1,14 @@
 ---
 # Barman S3 credentials for ha-pg.
 #
-# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
-# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
-# problem and the same note. The values are the same today, so this is correct
-# but coupled: rotating paperless's Hetzner credentials would silently change
-# the backup credentials for Home Assistant and Woodpecker too.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
-# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
-# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
-#
-#   bao kv put homelab/seed/home-assistant/hetzner-creds \
-#     @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
-#
-# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
-# 403 on `bao kv list seed`, which was a wrong-path error rather than a
-# permissions problem.
+# The earlier note here proposed a dedicated seed/home-assistant path. That was
+# the wrong direction: it would have made a fifth copy of the same key look like
+# a fourth independent credential. It did get one thing right, and it is worth
+# keeping: the 403 recorded in #397 was a wrong-path error, not a permissions
+# problem, because the mount is `homelab` and the path is homelab/seed/...
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -35,4 +28,4 @@ spec:
     name: hetzner-s3-creds
   dataFrom:
     - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/immich/hetzner-creds-externalsecret.yaml

@@ -15,4 +15,4 @@ spec:
     name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
   dataFrom:
     - extract:
-        key: seed/immich/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml

@@ -38,11 +38,15 @@ spec:
             summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
             description: >-
               The drive has used {{ $value }}% of its rated write endurance.
-              Nothing is failing and no data is at risk yet. Check what the
-              drive is carrying before deciding urgency: an OSD is replaceable
-              with a rebalance, whereas the EPHEMERAL volume has no such
-              escape — a node simply stops when it goes. The Disk Health
-              dashboard resolves which drive holds which role.
+              Nothing is failing and no data is at risk yet. What the drive
+              carries decides the urgency, and the Disk Health dashboard
+              resolves which role it holds. An OSD losing a drive costs a
+              rebalance. EPHEMERAL is Talos's /var and is meant to be
+              disposable — container images, kubelet state and pod scratch all
+              rebuild — so a worn drive there is a reasonable place for it. The
+              one thing on that volume which is not free to lose is the node's
+              etcd member data: quorum survives losing one of three, but the
+              member has to be removed and re-added.
         - alert: NvmeDriveWearCritical
           expr: smartctl_device_percentage_used >= 95
           for: 1h

kubernetes/apps/monitoring/dashboards/disk-health.json

@@ -70,7 +70,7 @@
     },
     {
       "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var, holding container images, pod scratch space and kubelet state. A full /var takes the node down, and unlike Ceph nothing rebalances away from it.",
+      "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var. Its contents are meant to be disposable: container images, kubelet state and pod scratch all rebuild. Filling it is still a problem, because a node with no room to write stops being useful long before anything is lost, and nothing rebalances /var the way Ceph rebalances a full OSD.",
       "fieldConfig": {
         "defaults": {
           "color": {"mode": "thresholds"},
@@ -453,7 +453,7 @@
       "id": 8,
       "options": {
         "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
-        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, so the drive with the least endurance left can be the one carrying container state.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
+        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, and putting a worn one on EPHEMERAL is a sensible destination rather than a hazard — that volume is built to be rebuilt. Check the inventory table for what a given drive holds before reading its wear as urgent.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
         "mode": "markdown"
       },
       "title": "Gaps and caveats",

kubernetes/apps/paperless/hetzner-creds-externalsecret.yaml

@@ -1,3 +1,27 @@
+# CANONICAL NOTE on the Hetzner S3 credentials. The other consumers point here
+# rather than repeating it.
+#
+# There is exactly ONE Hetzner S3 key, and every barman ObjectStore in this
+# cluster uses it. It used to live under four paths — seed/paperless,
+# seed/authentik, seed/immich, seed/trmnl — with byte-identical contents,
+# verified by comparing SHA-256 fingerprints of the synced Secrets rather than
+# by reading them. The same key also sits in infrastructure/velero-hetzner, in
+# a different shape (a single `cloud` field holding an AWS credentials file);
+# its access key id fingerprints the same.
+#
+# Per-app paths implied a separation that did not exist. Rotating the key at
+# Hetzner changes it for everyone regardless of how many paths hold a copy, so
+# the separate paths bought nothing and cost a standing invitation to believe
+# otherwise — #397 was opened on exactly that misreading, proposing to give
+# woodpecker a fifth copy.
+#
+# Real separation needs one Hetzner key per consumer, and Hetzner keys can only
+# be created by hand in their console. Until someone wants that, one path is the
+# honest representation.
+#
+# Not yet folded in: velero. It needs the credentials file form, so moving it
+# here means templating the INI from these fields — worth doing, not worth
+# bundling with a path swap that touches every database backup.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
@@ -15,4 +39,4 @@ spec:
     name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
   dataFrom:
     - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/trmnl/hetzner-creds-externalsecret.yaml

@@ -12,4 +12,4 @@ spec:
     name: hetzner-s3-creds
   dataFrom:
     - extract:
-        key: seed/trmnl/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/woodpecker/hetzner-creds-externalsecret.yaml

@@ -1,11 +1,8 @@
 ---
 # Barman S3 credentials for wp-pg.
 #
-# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
-# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
-# that path needs OpenBao write access (`bao kv list seed` → 403 for the
-# identities available here), so it stays open. The values are identical, so it
-# is a one-line swap of the `key:` below once the path exists.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -24,4 +21,4 @@ spec:
     name: hetzner-s3-creds
   dataFrom:
     - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3

kubernetes/apps/wordpress/externalsecret.yaml

@@ -36,11 +36,9 @@ spec:
   target:
     name: hetzner-s3-creds
   dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration — same approach woodpecker took. FOLLOW-UP: give
-    # wordpress its own seed/wordpress/hetzner-creds path once OpenBao can be
-    # written, and repoint this here (the value is the same, so it's a drop-in
-    # swap).
+    # One shared path for one shared credential. There is a single Hetzner S3
+    # key behind all of these — see the note in
+    # ../paperless/hetzner-creds-externalsecret.yaml for why per-app paths were
+    # the wrong shape and what would have to change for them to be right.
     - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3