fix: [alert] NvmeDriveWearHigh (monitoring) (#675 )

Merge pull request 'refactor(secrets): drop the unused barman-cloud-creds' (#677 ) from drop-unused-barman-creds into master
Reviewed-on: k.renner/homelab#677
2026-08-27 19:45:14 +00:00 · 2026-08-27 19:20:01 +00:00 · 2026-08-27 21:18:08 +02:00 · 2026-08-27 19:04:23 +00:00 · 2026-08-27 21:02:28 +02:00 · 2026-08-27 18:38:33 +00:00
14 changed files with 63 additions and 89 deletions
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: authentik
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->authentik-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/authentik/barman-creds
@@ -15,4 +15,4 @@ spec:
    name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
  dataFrom:
    - extract:
-        key: seed/authentik/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -27,5 +27,4 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
@@ -1,21 +1,14 @@
 ---
 # Barman S3 credentials for ha-pg.
 #
-# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
-# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
-# problem and the same note. The values are the same today, so this is correct
-# but coupled: rotating paperless's Hetzner credentials would silently change
-# the backup credentials for Home Assistant and Woodpecker too.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
-# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
-# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
-#
-#   bao kv put homelab/seed/home-assistant/hetzner-creds \
-#     @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
-#
-# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
-# 403 on `bao kv list seed`, which was a wrong-path error rather than a
-# permissions problem.
+# The earlier note here proposed a dedicated seed/home-assistant path. That was
+# the wrong direction: it would have made a fifth copy of the same key look like
+# a fourth independent credential. It did get one thing right, and it is worth
+# keeping: the 403 recorded in #397 was a wrong-path error, not a permissions
+# problem, because the mount is `homelab` and the path is homelab/seed/...
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -35,4 +28,4 @@ spec:
    name: hetzner-s3-creds
  dataFrom:
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: immich
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->immich-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/immich/barman-creds
@@ -15,4 +15,4 @@ spec:
    name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
  dataFrom:
    - extract:
-        key: seed/immich/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -13,6 +13,5 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
  - pvcs.yaml
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: paperless
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->paperless-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/paperless/barman-creds
@@ -1,3 +1,27 @@
+# CANONICAL NOTE on the Hetzner S3 credentials. The other consumers point here
+# rather than repeating it.
+#
+# There is exactly ONE Hetzner S3 key, and every barman ObjectStore in this
+# cluster uses it. It used to live under four paths — seed/paperless,
+# seed/authentik, seed/immich, seed/trmnl — with byte-identical contents,
+# verified by comparing SHA-256 fingerprints of the synced Secrets rather than
+# by reading them. The same key also sits in infrastructure/velero-hetzner, in
+# a different shape (a single `cloud` field holding an AWS credentials file);
+# its access key id fingerprints the same.
+#
+# Per-app paths implied a separation that did not exist. Rotating the key at
+# Hetzner changes it for everyone regardless of how many paths hold a copy, so
+# the separate paths bought nothing and cost a standing invitation to believe
+# otherwise — #397 was opened on exactly that misreading, proposing to give
+# woodpecker a fifth copy.
+#
+# Real separation needs one Hetzner key per consumer, and Hetzner keys can only
+# be created by hand in their console. Until someone wants that, one path is the
+# honest representation.
+#
+# Not yet folded in: velero. It needs the credentials file form, so moving it
+# here means templating the INI from these fields — worth doing, not worth
+# bundling with a path swap that touches every database backup.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
@@ -15,4 +39,4 @@ spec:
    name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
  dataFrom:
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -20,7 +20,6 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
  - pvcs.yaml
 configMapGenerator:
@@ -12,4 +12,4 @@ spec:
    name: hetzner-s3-creds
  dataFrom:
    - extract:
-        key: seed/trmnl/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -1,11 +1,8 @@
 ---
 # Barman S3 credentials for wp-pg.
 #
-# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
-# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
-# that path needs OpenBao write access (`bao kv list seed` → 403 for the
-# identities available here), so it stays open. The values are identical, so it
-# is a one-line swap of the `key:` below once the path exists.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -24,4 +21,4 @@ spec:
    name: hetzner-s3-creds
  dataFrom:
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -36,11 +36,9 @@ spec:
  target:
    name: hetzner-s3-creds
  dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration — same approach woodpecker took. FOLLOW-UP: give
-    # wordpress its own seed/wordpress/hetzner-creds path once OpenBao can be
-    # written, and repoint this here (the value is the same, so it's a drop-in
-    # swap).
+    # One shared path for one shared credential. There is a single Hetzner S3
+    # key behind all of these — see the note in
+    # ../paperless/hetzner-creds-externalsecret.yaml for why per-app paths were
+    # the wrong shape and what would have to change for them to be right.
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -69,6 +69,25 @@ cluster:
 # which is the constraint that makes k8s-2 and k8s-3 trip DiskPressure whenever
 # they absorb another node's pods.
 #
+# One caveat this node does NOT share with k8s-2 and k8s-3: their P3s report
+# zero media errors, this one reports 18 (smartctl_device_media_errors,
+# verified 2026-08-27 — the only non-zero counter in the fleet). Everything
+# else on the drive is clean, available spare 100% and critical-warning byte
+# 0, and NvmeDriveMediaErrors alerts on the increase rather than the total, so
+# a static count never surfaces there.
+#
+# Whether those 18 are old damage from its OSD career or ongoing is NOT
+# established. Prometheus only has a few hours of history for this drive, flat
+# at 18, because the exporter first scraped this node at its conversion — the
+# series being young is not evidence the count is stable. Settle it on the
+# node with `smartctl -a /dev/nvme1n1` and compare the error-log entries
+# against power-on hours.
+#
+# That question matters more here than the wear number does, because
+# /var/lib/etcd lives on EPHEMERAL: if the count is climbing, this node's etcd
+# member data is sitting on failing media and EPHEMERAL belongs on the system
+# disk instead — which means a wipe and install media, per the note below.
+#
 # THIS ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
 # solely when the volume does not yet exist; a config applied after EPHEMERAL
 # has been created elsewhere is silently ignored, and relocating it then
Author	SHA1	Message	Date
issue-fix-bot	bdffe1804b	fix: [alert] NvmeDriveWearHigh (monitoring) (#675 ) ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details	2026-08-27 19:45:14 +00:00
k.renner	c6b1aa109b	Merge pull request 'refactor(secrets): drop the unused barman-cloud-creds' (#677 ) from drop-unused-barman-creds into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#677	2026-08-27 19:20:01 +00:00
claude-bot	2f3d952656	refactor(secrets): drop the unused barman-cloud-creds ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details Three ExternalSecrets synced a Secret that nothing reads. Established rather than assumed, from three directions: every live ObjectStore names hetzner-s3-creds, no CNPG Cluster uses the legacy inline barmanObjectStore (all seven run the barman-cloud plugin against an ObjectStore CR), and no pod in the cluster mounts barman-cloud-creds. They are almost certainly left from before the plugin migration, when the inline configuration read a Secret of that name. Worth stating plainly because it is not merely untidy: seed/{authentik,immich, paperless}/barman-creds hold a Hetzner S3 credential that is NOT the one in use. Fingerprint c2c5cf3e against 87954b64 for the live hetzner-s3 path, with the three copies identical to each other. A valid key with no consumer is exposure without benefit, so if it is a distinct S3 key it wants revoking at Hetzner, which only a human can do — deleting our copies does not. Removing the manifests does NOT remove the objects. All three carry kustomize.toolkit.fluxcd.io/prune: disabled from the seed->app-KS handoffs, so Flux skips them during garbage collection by design. The ExternalSecrets, their Secrets and the three OpenBao paths have to be deleted explicitly after this merges; git alone leaves them live and orphaned, which is worse than leaving them declared. The OpenBao paths deliberately stay until then. Deleting them first would leave three ExternalSecrets failing to sync against a key that no longer exists, for no gain.	2026-08-27 21:18:08 +02:00
k.renner	0a87f91116	Merge pull request 'refactor(secrets): one OpenBao path for the one Hetzner S3 credential' (#676 ) from hetzner-creds-shared-path into master ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#676	2026-08-27 19:04:23 +00:00
claude-bot	c59f0050b3	refactor(secrets): one OpenBao path for the one Hetzner S3 credential ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details #397 asked for the opposite: give woodpecker its own path so a rotation of paperless's credentials would stop silently changing woodpecker's. The premise does not hold. There is one Hetzner S3 key, and rotating it at Hetzner changes it for every consumer no matter how many paths hold a copy. A fifth copy would have bought nothing and made the separation look real. Seven ExternalSecrets were reading it from four paths — seed/paperless (four namespaces), seed/authentik, seed/immich, seed/trmnl. They now all read infrastructure/hetzner-s3, which sits beside the existing infrastructure/velero-hetzner. That the four paths are byte-identical was verified rather than assumed, and without reading them: SHA-256 over the three fields of each synced Secret returns the same fingerprint in all seven namespaces. The same check says infrastructure/velero-hetzner carries the same access key id, in a different shape — a single `cloud` field holding an AWS credentials file. Velero is deliberately not folded in. Moving it means templating that INI from these fields, which is worth doing and not worth bundling with a path swap that touches every database backup at once. The reasoning lives once, in the paperless manifest, with the others pointing at it. Repeating it six times is how six copies drift. Kept from the note this replaces, because it was right and cost someone an afternoon: the 403 recorded in #397 was a wrong-path error, not missing permissions. The mount is `homelab`, so the path is homelab/seed/... — `bao kv list homelab/seed` works with the identity available here. homelab/infrastructure/hetzner-s3 now exists, copied from the paperless path inside OpenBao so the value never left it, and fingerprints identical to its source. Writes turned out to be possible all along: they fail through openbao.renner.uno with a response the client cannot parse, and succeed against the Service. That is worth knowing before the next person concludes, as I first did, that the identity is read-only. Closes #397	2026-08-27 21:02:28 +02:00
k.renner	48c46610a6	Merge pull request 'docs(monitoring): stop treating a worn EPHEMERAL drive as a hazard' (#674 ) from ephemeral-framing into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#674	2026-08-27 18:38:33 +00:00