fix: [alert] NvmeDriveWearHigh (monitoring) (#675 )

Merge pull request 'refactor(secrets): drop the unused barman-cloud-creds' (#677 ) from drop-unused-barman-creds into master
Reviewed-on: k.renner/homelab#677
2026-08-27 19:45:14 +00:00 · 2026-08-27 19:20:01 +00:00 · 2026-08-27 21:18:08 +02:00 · 2026-08-27 19:04:23 +00:00
7 changed files with 19 additions and 57 deletions
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: authentik
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->authentik-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/authentik/barman-creds
@@ -27,5 +27,4 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: immich
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->immich-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/immich/barman-creds
@@ -13,6 +13,5 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
  - pvcs.yaml
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: paperless
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->paperless-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/paperless/barman-creds
@@ -20,7 +20,6 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
  - pvcs.yaml
 configMapGenerator:
@@ -69,6 +69,25 @@ cluster:
 # which is the constraint that makes k8s-2 and k8s-3 trip DiskPressure whenever
 # they absorb another node's pods.
 #
+# One caveat this node does NOT share with k8s-2 and k8s-3: their P3s report
+# zero media errors, this one reports 18 (smartctl_device_media_errors,
+# verified 2026-08-27 — the only non-zero counter in the fleet). Everything
+# else on the drive is clean, available spare 100% and critical-warning byte
+# 0, and NvmeDriveMediaErrors alerts on the increase rather than the total, so
+# a static count never surfaces there.
+#
+# Whether those 18 are old damage from its OSD career or ongoing is NOT
+# established. Prometheus only has a few hours of history for this drive, flat
+# at 18, because the exporter first scraped this node at its conversion — the
+# series being young is not evidence the count is stable. Settle it on the
+# node with `smartctl -a /dev/nvme1n1` and compare the error-log entries
+# against power-on hours.
+#
+# That question matters more here than the wear number does, because
+# /var/lib/etcd lives on EPHEMERAL: if the count is climbing, this node's etcd
+# member data is sitting on failing media and EPHEMERAL belongs on the system
+# disk instead — which means a wipe and install media, per the note below.
+#
 # THIS ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
 # solely when the volume does not yet exist; a config applied after EPHEMERAL
 # has been created elsewhere is silently ignored, and relocating it then
Author	SHA1	Message	Date
issue-fix-bot	bdffe1804b	fix: [alert] NvmeDriveWearHigh (monitoring) (#675 ) ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details	2026-08-27 19:45:14 +00:00
k.renner	c6b1aa109b	Merge pull request 'refactor(secrets): drop the unused barman-cloud-creds' (#677 ) from drop-unused-barman-creds into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#677	2026-08-27 19:20:01 +00:00
claude-bot	2f3d952656	refactor(secrets): drop the unused barman-cloud-creds ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details Three ExternalSecrets synced a Secret that nothing reads. Established rather than assumed, from three directions: every live ObjectStore names hetzner-s3-creds, no CNPG Cluster uses the legacy inline barmanObjectStore (all seven run the barman-cloud plugin against an ObjectStore CR), and no pod in the cluster mounts barman-cloud-creds. They are almost certainly left from before the plugin migration, when the inline configuration read a Secret of that name. Worth stating plainly because it is not merely untidy: seed/{authentik,immich, paperless}/barman-creds hold a Hetzner S3 credential that is NOT the one in use. Fingerprint c2c5cf3e against 87954b64 for the live hetzner-s3 path, with the three copies identical to each other. A valid key with no consumer is exposure without benefit, so if it is a distinct S3 key it wants revoking at Hetzner, which only a human can do — deleting our copies does not. Removing the manifests does NOT remove the objects. All three carry kustomize.toolkit.fluxcd.io/prune: disabled from the seed->app-KS handoffs, so Flux skips them during garbage collection by design. The ExternalSecrets, their Secrets and the three OpenBao paths have to be deleted explicitly after this merges; git alone leaves them live and orphaned, which is worse than leaving them declared. The OpenBao paths deliberately stay until then. Deleting them first would leave three ExternalSecrets failing to sync against a key that no longer exists, for no gain.	2026-08-27 21:18:08 +02:00
k.renner	0a87f91116	Merge pull request 'refactor(secrets): one OpenBao path for the one Hetzner S3 credential' (#676 ) from hetzner-creds-shared-path into master ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#676	2026-08-27 19:04:23 +00:00