Merge pull request 'fix(bootstrap): portable DR secret commands, visible YubiKey prompts' (#1482) from dr-secrets-age-fixes into main

Reviewed-on: #1482

kubernetes/MIGRATION-sops-openbao.md

@@ -52,9 +52,9 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key
 
 **Unseal is automatic since 2026-09-08** (static seal, PR #825, issue #805):
 the seal key lives in the `openbao-seal-key` Secret (created by `bootstrap.sh`
-from `pass show homelab/openbao/seal-key`) and every restarted replica unseals
-itself. The shamir key above became the **recovery key** in the migration; it
-is kept in `pass` under its old name, `homelab/openbao/unseal-key`. The server
+from `~/.config/homelab-secrets/seal-key.age`) and every restarted replica
+unseals itself. The shamir key above became the **recovery key** in the
+migration; it is `~/.config/homelab-secrets/openbao-recovery-key.age`. The server
 runs three raft peers (see `infrastructure/openbao/helmrelease.yaml`); a sealed
 one is neither a voter nor a standby, so a replica that stays sealed after a
 restart — wrong or missing seal-key Secret — costs redundancy, two cost
@@ -237,19 +237,21 @@ one full reconcile + one OpenBao pod restart/unseal cycle:
 - **Backups:** ✅ nightly raft-snapshot CronJob (03:00, keeps 7 on a
   dedicated PVC; see infrastructure/openbao/snapshot-cronjob.yaml) + velero
   `openbao-daily` Schedule (06:15) ships that PVC offsite, 30 d retention.
-  A raft snapshot restore needs the matching seal key — which lives in
-  `pass`, outside the cluster (GPG key backed up externally 2026-07-14).
+  A raft snapshot restore needs the matching seal key — which lives outside
+  the cluster in `~/.config/homelab-secrets/seal-key.age` (age, encrypted to
+  both YubiKeys and the offline paper key; copy on nas-2).
   A failed snapshot Job is the sealed-after-restart alarm. Still owed:
   one tested restore before phase 4.
-- **Cluster rebuild order:** seal-key Secret from `pass` → Flux brings up
+- **Cluster rebuild order:** seal-key Secret from `seal-key.age` → Flux brings up
   openbao (empty) → init → restore raft snapshot → replicas unseal themselves
   → ClusterSecretStore Ready → ESO materializes Secrets → apps start
   (`bootstrap.sh flux` does all of it; RUNBOOK-cold-rebuild.md Step 4 by
   hand). Until then, secret-consuming apps simply stay Pending on missing
   Secrets — same as any missing dependency.
-- **Break-glass:** `pass` holds seal key, recovery key + root token (GPG,
-  offline-capable). With those, everything is recoverable from a raft
-  snapshot alone.
+- **Break-glass:** `~/.config/homelab-secrets` holds the seal key and
+  recovery key (YubiKeys + paper key, copy on nas-2);
+  `~/.config/openbao/root-token.age` holds the root token (YubiKeys). With
+  those, everything is recoverable from a raft snapshot alone.
 - **Ansible Vault password:** `ansible/vault-password.sh` reads
   `ANSIBLE_VAULT_PASSWORD` from the environment with no offline backup as of
   2026-07-14. It gates decryption of several Ansible-Vault files (PBS

kubernetes/RUNBOOK-cold-rebuild.md

@@ -67,17 +67,24 @@ These must exist outside the cluster or the rebuild cannot complete:
       `flux` phase creates the `sops-age` Secret, and any not-yet-migrated sops
       files decrypt with it. (Post-OpenBao-phase-4 this may be retired — see
       `MIGRATION-sops-openbao.md`.)
-- [ ] **OpenBao seal key, recovery key + root token** — in the GPG `pass`
-      store (`homelab/openbao/seal-key`, `homelab/openbao/unseal-key` — the
-      former shamir key, now the recovery key — and `homelab/openbao/root-token`),
-      and the GPG private key backed up off-machine. The seal key is the one
-      the snapshots are encrypted under: without it, no OpenBao snapshot can be
-      restored and no secret-consuming app can start.
+- [ ] **OpenBao seal key, recovery key, restore + pull credentials** — age
+      files in `~/.config/homelab-secrets/` (`seal-key.age`,
+      `openbao-recovery-key.age` — the former shamir key —, `s3-access.age`,
+      `s3-secret.age`, `ghcr-pull.age`, `hetzner-cloud-api-secret.age`),
+      encrypted to both YubiKeys and the
+      offline paper key in the safe; a copy is on nas-2 in
+      `~ladmin/homelab-secrets-backup` (`~/.ssh/ssh-keys-rotate dr-copy`). The
+      root token is `~/.config/openbao/root-token.age` (YubiKeys only). Needed
+      to decrypt: one YubiKey with its PIV PIN, or the paper key. The seal key
+      is the one the snapshots are encrypted under: without it, no OpenBao
+      snapshot can be restored and no secret-consuming app can start.
 - [ ] **OpenBao raft snapshot** reachable — nightly PVC snapshots
       (`infrastructure/openbao/snapshot-cronjob.yaml`) plus the Velero
       `openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
       copy is the one that survives.
-- [ ] **Tooling:** `talosctl`, `kubectl`, `flux`, `tofu`, `bao`, plus the
+- [ ] **Tooling:** `talosctl`, `kubectl`, `flux`, `tofu`, `bao`, `age` +
+      `age-plugin-yubikey` (the plugin is only needed for the YubiKeys; the
+      paper key needs `age` alone), plus the
       OpenTofu backend + UniFi credentials the `opentofu/` stack needs. No
       Proxmox credentials — there is no Proxmox any more.
 - [ ] **Install media** written and to hand: the Image Factory ISO named in
@@ -299,14 +306,16 @@ output** — a green step over a still-forming Ceph is a known trap.
 ## Step 4 — openbao: restore the snapshot
 
 `bootstrap.sh flux` does this step itself (`restore_openbao`): it creates the
-`openbao-seal-key` Secret from `pass` before Flux brings OpenBao up, initialises
+`openbao-seal-key` Secret from `~/.config/homelab-secrets/seal-key.age` before Flux brings OpenBao up, initialises
 the empty server, restores the newest raft snapshot straight out of the Velero
 kopia repository, and waits until every replica has unsealed itself from the
 static seal. What follows is the same procedure by hand, for when the script
 is not an option.
 
 OpenBao unseals itself from the static seal (`openbao-seal-key` Secret, key
-`key`, the 32-byte base64 key from `pass show homelab/openbao/seal-key`). The
+`key`, the 32-byte base64 key in `~/.config/homelab-secrets/seal-key.age`,
+age-encrypted to the owner's YubiKeys and the offline paper key; a copy sits on
+nas-2 in `~ladmin/homelab-secrets-backup`). The
 snapshots are encrypted under that key, so it must be the same key the backed
 up cluster used — a different one restores nothing. Until the snapshot is
 restored, the `ClusterSecretStore` stays NotReady, ESO cannot materialize
@@ -316,7 +325,8 @@ Secrets, and every secret-consuming app stays Pending.
 # 1. The seal key, before the pods start (a pod without it never starts).
 kubectl create namespace openbao --dry-run=client -o yaml | kubectl apply -f -
 kubectl -n openbao create secret generic openbao-seal-key \
-  --from-literal=key="$(pass show homelab/openbao/seal-key)"
+  --from-literal=key="$(age -d -i ~/.config/age/yubikeys.txt ~/.config/homelab-secrets/seal-key.age)"
+# Both YubiKeys lost: -i <file holding the paper key> instead.
 
 # 2. Wait for openbao-0 (uninitialised: 0/1 Ready is expected), then
 #    initialise it. With an auto-seal, init hands out recovery keys and the
@@ -337,7 +347,8 @@ kubectl -n openbao exec -it openbao-0 -- \
 
 # 4. The server seals against the snapshot's keyring and unseals itself from
 #    the static seal; openbao-1 and openbao-2 follow (OrderedReady). Nothing
-#    to type. Afterwards the tokens and recovery key from pass apply again.
+#    to type. Afterwards the operator's root token
+#    (~/.config/openbao/root-token.age) and recovery key apply again.
 kubectl -n openbao get pods -w
 ```
 
@@ -352,7 +363,7 @@ kubectl -n <ns> get externalsecret                # want: SecretSynced
 > A replica that stays **sealed** after a restart could not use the static
 > seal — the Secret is missing or does not match the key the data was
 > written under. Fix the Secret and restart the pod; `bao operator unseal`
-> with the recovery key (`pass show homelab/openbao/unseal-key`) does not
+> with the recovery key (`~/.config/homelab-secrets/openbao-recovery-key.age`) does not
 > unseal an auto-sealed server and is only for recovery-mode operations.
 > Details + rationale: `MIGRATION-sops-openbao.md`.
 
@@ -434,7 +445,7 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
   Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
   Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
   Its `flux` phase carries the OpenBao restore (Step 4); the seal key and the
-  other secrets are read from `pass` or streamed in at run time and never live
+  other secrets are decrypted from `~/.config/homelab-secrets` or streamed in at run time and never live
   in CI.
 - **OpenBao unseals itself, but only under the seal key the snapshot was
   written with.** Three raft replicas mean the service survives losing one; a

kubernetes/apps/monitoring-rules/prometheusrule-openbao.yaml

@@ -49,7 +49,7 @@ spec:
               failure would cost quorum. Check `kubectl -n openbao get pods`,
               the pod log and `bao status` in the not-ready pod. If it is
               sealed, the static seal failed: verify the openbao-seal-key
-              Secret matches `pass show homelab/openbao/seal-key`.
+              Secret matches ~/.config/homelab-secrets/seal-key.age (age, YubiKey).
 
         - alert: OpenBaoQuorumLost
           # Raft quorum of a three-node cluster is two. At one ready replica
@@ -76,7 +76,7 @@ spec:
               surface only when something rotates or a new secret is needed.
               Replicas unseal themselves from the static seal; one that stays
               sealed could not. Check its log and the openbao-seal-key Secret
-              against `pass show homelab/openbao/seal-key`, fix the Secret and
+              against ~/.config/homelab-secrets/seal-key.age, fix the Secret and
               restart the pod. The recovery key cannot unseal an auto-sealed
               server. Procedure: kubernetes/RUNBOOK-cold-rebuild.md, Step 4.
 

kubernetes/bootstrap.sh

@@ -35,7 +35,8 @@ set -euo pipefail
 # The mirror is kept current server-side by Gitea's native push-mirror. Full
 # cold-rebuild procedure: kubernetes/RUNBOOK-cold-rebuild.md.
 #
-# Requires: talosctl, kubectl, flux, sops age key at ~/.config/sops/age/keys.txt
+# Requires: talosctl, kubectl, flux, sops age key at ~/.config/sops/age/keys.txt,
+# age + age-plugin-yubikey (the DR secrets in ~/.config/homelab-secrets)
 
 TALOS_DIR="$(cd "$(dirname "$0")/talos" && pwd)"
 REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
@@ -190,13 +191,16 @@ if [[ "$FLUX_SOURCE" == "restore-test" && "$PHASE" != "flux" && "$PHASE" != "wai
 fi
 OPENBAO_NS="openbao"
 
-# Where the secrets come from. On a workstation, the operator's pass store. In
-# the in-cluster Job there is no pass and no GPG key — and there should not be:
-# the pass store opens everything, and it stays on the operator's machine. Set
-# OPENBAO_SECRETS_DIR and the run reads them from files instead, which the
+# Where the secrets come from. On a workstation, age files in
+# HOMELAB_SECRETS_DIR, encrypted to the operator's YubiKeys and an offline paper
+# key (a copy sits on nas-2). HOMELAB_AGE_IDENTITY picks what decrypts them: the
+# YubiKey identity stubs by default (touch per file), the paper key's file when
+# both sticks are gone. In the in-cluster Job there is neither — and there
+# should not be: those keys open everything, and they stay with the operator.
+# Set OPENBAO_SECRETS_DIR and the run reads them from files instead, which the
 # operator streams in when the run asks for them:
 #
-#     pass show homelab/openbao/seal-key \
+#     age -d -i ~/.config/age/yubikeys.txt ~/.config/homelab-secrets/seal-key.age \
 #       | kubectl -n vmtest exec -i <pod> -- sh -c 'cat > /secrets/seal-key'
 #
 # They exist in the pod's filesystem for the length of the run and are never
@@ -240,6 +244,15 @@ git_token_for_source() {
     fi
 }
 
+# Literal ~ on purpose: the commands are eval'd on the workstation, which
+# expands it there, and the in-cluster Job prints them for the operator to run
+# on that workstation, where the Job's own HOME (/work) would be wrong.
+# shellcheck disable=SC2088
+[[ -n "${HOMELAB_SECRETS_DIR:-}" ]] || HOMELAB_SECRETS_DIR='~/.config/homelab-secrets'
+# shellcheck disable=SC2088
+[[ -n "${HOMELAB_AGE_IDENTITY:-}" ]] || HOMELAB_AGE_IDENTITY='~/.config/age/yubikeys.txt'
+AGE_SECRET="age -d -i $HOMELAB_AGE_IDENTITY $HOMELAB_SECRETS_DIR"
+
 # Format: <file in SECRETS_DIR>|<what it is>|<how the operator produces it>
 OPENBAO_REQUIRED_SECRETS=(
     "sops-age.key|SOPS age key|cat ~/.config/sops/age/keys.txt"
@@ -255,18 +268,18 @@ OPENBAO_REQUIRED_SECRETS=(
     # Emptiness is not tested here on purpose — require_secrets rejects an empty
     # file and names what is missing.
     "git-token|token for flux bootstrap, for whichever forge FLUX_SOURCE names|git_token_for_source"
-    "s3-access|Hetzner restore access key|pass show homelab/hetztner/k8s-restore-access"
-    "s3-secret|Hetzner restore secret key|pass show homelab/hetztner/k8s-restore-secret"
+    "s3-access|Hetzner restore access key|$AGE_SECRET/s3-access.age"
+    "s3-secret|Hetzner restore secret key|$AGE_SECRET/s3-secret.age"
     # The static seal key. The snapshots are encrypted under it, so it has to
     # be the one the backed-up cluster ran with — a fresh key restores nothing.
-    "seal-key|OpenBao seal key|pass show homelab/openbao/seal-key"
+    "seal-key|OpenBao seal key|$AGE_SECRET/seal-key.age"
     # Read by the node registry auth (see registry_auth_patch): the images an app
     # needs while the cluster is still being built are mirrored to a private
     # ghcr.io namespace, and a node has to be able to pull them before anything
     # in the cluster exists to hold a credential. Demanded up front like the
     # rest, because a rebuild that discovers it missing has already wiped the
     # disks.
-    "ghcr-pull|GHCR pull token for the image mirror|pass show homelab/github/k8s-package-pull"
+    "ghcr-pull|GHCR pull token for the image mirror|$AGE_SECRET/ghcr-pull.age"
 )
 # Asked for by name only, never by a bare require_secrets: a rebuild of the
 # real cluster cannot depend on the OpenBao it is rebuilding. Scoped to the
@@ -281,8 +294,8 @@ HCLOUD_REQUIRED_SECRETS=(
 # during a real recovery or in the in-cluster Job.
 #
 # The only difference between those two is where the material comes from, and it
-# is confined to this function: on a laptop the commands in the list produce it
-# from pass; in the Job the operator streams it in. Everything downstream cannot
+# is confined to this function: on a laptop the commands in the list decrypt
+# it from HOMELAB_SECRETS_DIR; in the Job the operator streams it in. Everything downstream cannot
 # tell the difference — which is the point, because the recovery path must not be
 # a path that only gets exercised during a recovery.
 # require_secrets [name...]
@@ -311,7 +324,7 @@ require_secrets() {
         wanted=("${OPENBAO_REQUIRED_SECRETS[@]}")
     fi
 
-    # Which branch: fetch from pass, or wait for someone to stream files in.
+    # Which branch: decrypt locally, or wait for someone to stream files in.
     #
     # The condition cannot be "is OPENBAO_SECRETS_DIR set", because THIS
     # FUNCTION sets it. Once a subset call had run — hcloudtest asks for
@@ -322,7 +335,7 @@ require_secrets() {
     # maintenance.
     #
     # SECRETS_SELF_FETCH records which world we are in, and it is decided once:
-    # an empty OPENBAO_SECRETS_DIR on entry means pass, an inherited one means
+    # an empty OPENBAO_SECRETS_DIR on entry means local age decryption, an inherited one means
     # the Job. Later calls follow that decision instead of re-deriving it from
     # state this function has since changed.
     if [[ -z "$OPENBAO_SECRETS_DIR" || "${SECRETS_SELF_FETCH:-0}" == "1" ]]; then
@@ -342,12 +355,14 @@ require_secrets() {
             name=${entry%%|*}; what=${entry#*|}; how=${what#*|}; what=${what%%|*}
             # Already collected by an earlier call — a subset call fetches only
             # what it asked for, so the full call that follows must fill the
-            # gaps rather than re-run every command. `pass show` on a locked
-            # store prompts, and prompting twice for the same key is how an
+            # gaps rather than re-run every command. Each decryption waits for
+            # a YubiKey touch, and asking twice for the same key is how an
             # unattended run turns into a hung one.
             [[ -s "$OPENBAO_SECRETS_DIR/$name" ]] && continue
-            if ! eval "$how" > "$OPENBAO_SECRETS_DIR/$name" 2>"$OPENBAO_SECRETS_DIR/.err"; then
-                echo; cat "$OPENBAO_SECRETS_DIR/.err" >&2
+            # stderr stays on the terminal: age-plugin-yubikey asks for the
+            # PIN and the touch there, and a captured prompt is a silent hang.
+            echo
+            if ! eval "$how" > "$OPENBAO_SECRETS_DIR/$name"; then
                 fail "could not obtain the $what with: $how"
             fi
             [[ -s "$OPENBAO_SECRETS_DIR/$name" ]] \
@@ -382,7 +397,7 @@ OPENBAO_NS_HINT=vmtest
 
 # Reads one secret by its name in the list above. In the Job that is the file the
 # operator streamed in; on a workstation it runs the command the list names,
-# which for these is `pass show <path>`.
+# which for these is `age -d` of a file in HOMELAB_SECRETS_DIR.
 #
 # The list is the only place the paths appear. They used to sit here as
 # constants as well, which is how the same value ends up defined twice and one
@@ -1486,7 +1501,8 @@ fi
 
 # Before anything destructive, and after the vmtest block has decided which
 # phases will run. A rebuild that will need these has no business wiping disks
-# first, only to discover an hour later that pass is locked or a token is unset —
+# first, only to discover an hour later that no YubiKey is plugged in or a
+# token is unset —
 # by then the old cluster is gone and the new one cannot be finished.
 #
 # In the Job this is where the file-waiting branch blocks, which is the right
@@ -1994,7 +2010,8 @@ bao_exec() {
 # have been doing this all along.
 #
 # Reading them needs three things, all of which live OUTSIDE the cluster, which
-# is the point: bucket credentials from pass (their own key, not Velero's),
+# is the point: bucket credentials from HOMELAB_SECRETS_DIR (their own key, not
+# Velero's),
 # kopia, and the repository password. The last is not a secret — Velero
 # hardcodes `static-passw0rd` in its source. Worth knowing for what it means:
 # the offsite backup's confidentiality rests on bucket access alone.
@@ -2071,7 +2088,8 @@ restore_openbao() {
     # rather than an unseal key and the server unseals itself; the only thing
     # used from its output is the root token, for the next few seconds, and
     # then it stops existing: restoring replaces the keyring with the
-    # snapshot's, after which the root token and recovery key from pass apply.
+    # snapshot's, after which the operator's root token
+    # (~/.config/openbao/root-token.age) and recovery key apply.
     #
     # No -force. Without it OpenBao checks that the snapshot was written under
     # the seal key this server runs with, and refuses a mismatch by name
@@ -2355,7 +2373,7 @@ if phase_runs flux; then
     # Secret would leave every replica sealed for good at its next restart.
     # That is a migration, not a bootstrap step.
     #
-    # --from-literal, not --from-file: `pass show` ends its output with a
+    # --from-literal, not --from-file: the stored value ends with a
     # newline, the substitution strips it, and OpenBao rejects a key that is
     # not exactly 32 bytes of base64. The key sits in this process's argument
     # list for the length of the call, as it did in the pod's before.

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -80,6 +80,25 @@ spec:
           # community blocklists.
           - 162.55.50.97/32
           - 2a01:4f8:c2c:4579::1/128
+      postoverflows:
+        s01-whitelist:
+          # http-crawl-non_statics counts requests regardless of outcome, so a
+          # signed-in user of any image-heavy UI (Immich timeline, Music
+          # Assistant covers) trips it with a burst of 200s. Scanners trip it
+          # with 404s for paths that do not exist. Drop the overflow unless at
+          # least half of its requests were client errors (the alert carries
+          # the scenario's last 5 events, cache_size: 5). Only this scenario
+          # is affected; probing/sensitive-files/cve still fire on their own.
+          crawl-non-statics-success-whitelist.yaml: |
+            name: krenner/crawl-non-statics-success-whitelist
+            description: "Ignore http-crawl-non_statics bursts that mostly succeeded"
+            whitelist:
+              reason: "crawl burst of successful requests is a UI, not a scanner"
+              expression:
+                - >-
+                  evt.Overflow.Alert.GetScenario() == 'crowdsecurity/http-crawl-non_statics' &&
+                  count(evt.Overflow.Alert.Events, {.GetMeta('http_status') startsWith '4'}) * 2
+                  < len(evt.Overflow.Alert.Events)
 
     # mTLS between agents and the LAPI. Do not switch back to password auth:
     # the LAPI runs without persistence, so its machine-registration DB is