feat(monitoring): alert on any blackbox target without a rule of its own

Nine of the thirteen blackbox-http targets were probed, graphed and completely
unalerted: gitea, immich, auth, home, prometheus, alerts and dash, plus the two
internal /-/healthy probes that exist specifically to catch a dead app behind a
live Authentik gate. That is the shape that let file.renner.uno serve 502 for
months.

BlackboxProbeFailing covers everything except the instances that already have a
dedicated rule. Measured against the running Prometheus before committing: the
selector returns exactly those nine and excludes the six with bespoke handling,
and the full expression as YAML delivers it evaluates to zero results today.

The exclusion list and the dedicated rules are one invariant, stated in the
file: an instance belongs in the list only while a rule elsewhere covers it,
and a new dedicated rule has to add its instance. Wrong in one direction the
alert fires twice, wrong in the other it never fires. Severity is warning
because a catch-all cannot know what any given target means — anything
deserving critical earns its own rule, which is also what puts it in the list.
That is what stops the list from being arbitrary.

BlackboxProbesAbsent is the check on the checks. Every endpoint alert reads
probe_success; if the job stops producing series none of them can fire, and
that silence is indistinguishable from health.

Also corrects the comment in prometheusrule-hetzner-backup.yaml claiming CNPG
pod metrics are not scraped and a granular archiving alert is a follow-up. Both
have existed for a while: cnpg-podmonitors.yaml scrapes the instances and
CNPGWalArchiveBacklog reads that collector.

.woodpecker/install-garage-synology.yml

@@ -1,27 +0,0 @@
-when:
-  - event: deployment
-    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "install-garage-synology"'
-
-steps:
-  - name: run-playbook
-    image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
-    environment:
-      ANSIBLE_FORCE_COLOR: "true"
-      ANSIBLE_HOST_KEY_CHECKING: "false"
-      SSH_PRIVATE_KEY:
-        from_secret: ssh_private_key
-      VAULT_PASS:
-        from_secret: vault_pass
-      PROXMOX_TOKEN_ID:
-        from_secret: proxmox_token_id
-      PROXMOX_TOKEN_SECRET:
-        from_secret: proxmox_token_secret
-    commands:
-      - mkdir -p ~/.ssh
-      - echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
-      - chmod 600 ~/.ssh/id_ed25519
-      - 'printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config'
-      - cd ansible
-      - echo "$$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
-      - export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
-      - ansible-playbook -i inventory/ plays/install-garage-synology.yml -v

.woodpecker/security-scan.yml

@@ -40,12 +40,28 @@ steps:
         cat /tmp/images.txt
         echo "---"
         i=0
+        failed=0
+        failed_names=""
         while IFS= read -r img; do
           i=$$((i+1))
           out="trivy-image-$$i.json"
           echo "=== [$$i] $$img -> $$out ==="
-          trivy image --skip-db-update --scanners vuln --timeout 15m --severity CRITICAL,HIGH --ignore-unfixed --format json -o "$$out" "$$img" || true
+          # Keep going on a bad image so one unpullable reference cannot cost
+          # the scan of every image after it, but record that it happened.
+          if ! trivy image --skip-db-update --scanners vuln --timeout 15m --severity CRITICAL,HIGH --ignore-unfixed --format json -o "$$out" "$$img"; then
+            failed=$$((failed+1))
+            failed_names="$$failed_names $$img"
+          fi
         done < /tmp/images.txt
+        echo "--- scanned $$((i - failed)) of $$i images ---"
+        if [ "$$failed" -gt 0 ]; then
+          echo "NOT SCANNED:$$failed_names"
+          exit 1
+        fi
+    # Non-zero above turns this step red without failing the workflow, so the
+    # fix agent still runs on what was scanned. Coverage is reported, not
+    # enforced: an incomplete scan is visible in the step status instead of
+    # being indistinguishable from a clean one.
     failure: ignore
 
   - name: security-fix-agent

ansible/inventory/host_vars/nas-1.renner.uno/vars.yml

@@ -1,37 +0,0 @@
----
-# PBS Client configuration for nas-1
-pbs_backup_paths:
-  - /volume1/Documents
-  - /volume1/homes
-  - /volume1/Homeassistant Backup
-  - /volume1/Games
-  - /volume1/Media
-  - /volume1/Music
-  - /volume1/video
-  - /volume2/Backup
-  - /volume1/Movies
-
-  # Add more paths as needed:
-  # - /volume1/photos
-  # - /volume1/music
-
-# PBS Server connection
-pbs_server_host: "pbs2.renner.uno"
-pbs_server_port: 8007
-pbs_datastore: "pool"
-pbs_backup_id: "nas-1"
-
-# PBS Authentication - credentials in vault.yml
-pbs_username: "root@pam!nas-1"
-
-# Garage S3 (K8s backup target)
-garage_image: "dxflrs/garage:v2.3.0"
-garage_zone: "dsm"
-garage_capacity: "2T"
-
-# SSH connection - use password auth (key not supported by DSM)
-ansible_ssh_pass: "{{ nas_ssh_password }}"
-ansible_become_pass: "{{ nas_ssh_password }}"
-
-# Python 3.9+ from Synology package
-ansible_python_interpreter: /usr/local/bin/python3.9

ansible/inventory/host_vars/nas-1.renner.uno/vault.yml

@@ -1,27 +0,0 @@
-$ANSIBLE_VAULT;1.1;AES256
-34663462636437633437613863366266306334666635643839313737343739643864613136613938
-3138393966306561656366396465383539303137333036300a376237376636356230343834386362
-36636535396330333732383233323263613132333166343739396137636538653435336662353138
-6565643866623235610a316266383933303830323936303735623338623165386339383733373435
-62366461363833616438373237383132366438326364363462333265366533323861376630643365
-39343036343838333639363432643738666630363063323636646661313539643465366532363438
-33303837623234613434633563666562346536346435353461373536323737376233386163326537
-61383963343737656438356136613537623538333163343466343739346135333439616432663763
-36663962633037376561326434303035333938646436663832303432346365353439666435613337
-34363636326530333233313231343566306432376532396666643761633136316333376337666135
-65363338623035663438323338663863366362653131336434323865326530326534623631336263
-36633063383031386161363131333636393364623436613331353733313235383062656131313164
-63393665633361313961363637663237616364643236383835393736643937393564656238376362
-31393465383361326264376133393233616264663636383337363962356532613438663637616536
-33653137656335323763313461323633616635346232303062323062333766383931313965346361
-66656461313239633366636463633465636463663937393932636337643861383937663662306466
-32353166643832303234643264633062616330393963333034623432613730366339306637613239
-34626161663666316339323762373639353335653663353635366664643063393432623039306237
-38363161343930363763393364663364353735343236313665313461626165636135303839363164
-37336638396564326166376233353730346563656266323535666265323062646232323938353732
-32636235333839623166633761376334316434613235333730353439343262393537363461333238
-32653435623362666662336532643832356163326332636436346132643237643030343761663332
-36663530356539303636663438646430363335353135323932613863666264363533326137353164
-31663032346236616263343637396439663334396166656364623637666535386530383535663435
-66303037303265616634373763616437333835333136336563323339666563663931396133633131
-3963653266633535303630653335666166313336653831623362

ansible/inventory/pve_hosts.yml

@@ -8,6 +8,8 @@
 #   pve_cluster (jupiter/neptune/saturn)  run Talos on bare metal; no SSH at all
 #   authentik_db / authentik_app          authentik runs in-cluster (CNPG)
 #   grafana                               runs in-cluster
+#   synology_nas (nas-1)                  powered off; Garage and the PBS client
+#                                         it hosted are gone with it
 # Their hosts no longer resolve, except the three that resolve to Talos nodes.
 
 [netbird_server]
@@ -17,10 +19,6 @@ netbird.renner.uno ansible_user=root
 proxy-1.renner.uno ansible_user=root
 proxy-2.renner.uno ansible_user=root
 
-
-[synology_nas]
-nas-1.renner.uno ansible_user=ladmin ansible_python_interpreter=/usr/bin/python3
-
 # Every bare-metal edge thin client. Gets the host-generic roles only:
 # flash-wear handling, the log shipper, the MQTT bridge, the VLAN
 # sub-interface. Nothing here assumes any peripheral is attached.

ansible/plays/install-garage-synology.yml

@@ -1,313 +0,0 @@
----
-# Synology-specific playbook using raw module (Python 3.8 workaround)
-# Deploys Garage (https://garagehq.deuxfleurs.fr) as a single-node S3 target
-# for K8s backups (Velero + CNPG Barman Cloud Plugin).
-#
-# Required secrets in host_vars/nas-1.renner.uno/vault.yml:
-#   garage_rpc_secret:     "<openssl rand -hex 32>"
-#   garage_admin_token:    "<openssl rand -base64 32>"
-#   garage_metrics_token:  "<openssl rand -base64 32>"
-#
-# Post-install bootstrap (run once, record the returned secret keys):
-#   docker exec garaged /garage status
-#   docker exec garaged /garage bucket create k8s-velero
-#   docker exec garaged /garage bucket create k8s-cnpg
-#   docker exec garaged /garage bucket create k8s-secrets
-#   docker exec garaged /garage bucket create k8s-migration
-#   docker exec garaged /garage key create velero-key
-#   docker exec garaged /garage key create cnpg-key
-#   docker exec garaged /garage key create secrets-key
-#   docker exec garaged /garage key create migration-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-velero    --key velero-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-cnpg      --key cnpg-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-secrets   --key secrets-key
-#   docker exec garaged /garage bucket allow --read --write --owner k8s-migration --key migration-key
-- name: Install Garage S3 on Synology NAS
-  hosts: synology_nas
-  gather_facts: false
-
-  vars:
-    garage_data_dir: "/volume1/docker/garage"
-
-  tasks:
-    - name: Create Garage directory tree
-      ansible.builtin.raw: |
-        mkdir -p {{ garage_data_dir }}/config \
-                 {{ garage_data_dir }}/meta \
-                 {{ garage_data_dir }}/data
-      changed_when: true
-
-    # Secret writes use the same compare-then-write pattern as garage.toml so
-    # rotation is idempotent AND a real change can drive force-recreate below.
-    # Garage reads these files once at startup; without container restart, a
-    # rotated token leaves the running process holding the previous value.
-    - name: Write rpc_secret
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/rpc_secret.new << 'SECRET_EOF'
-        {{ garage_rpc_secret }}
-        SECRET_EOF
-        chmod 600 {{ garage_data_dir }}/config/rpc_secret.new
-        if [ ! -f {{ garage_data_dir }}/config/rpc_secret ] \
-        || ! cmp -s {{ garage_data_dir }}/config/rpc_secret.new {{ garage_data_dir }}/config/rpc_secret; then
-          mv {{ garage_data_dir }}/config/rpc_secret.new {{ garage_data_dir }}/config/rpc_secret
-          echo RPC_SECRET_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/rpc_secret.new
-        fi
-      register: rpc_secret_write
-      changed_when: "'RPC_SECRET_CHANGED' in rpc_secret_write.stdout"
-      no_log: true
-
-    - name: Write admin_token
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/admin_token.new << 'SECRET_EOF'
-        {{ garage_admin_token }}
-        SECRET_EOF
-        chmod 600 {{ garage_data_dir }}/config/admin_token.new
-        if [ ! -f {{ garage_data_dir }}/config/admin_token ] \
-        || ! cmp -s {{ garage_data_dir }}/config/admin_token.new {{ garage_data_dir }}/config/admin_token; then
-          mv {{ garage_data_dir }}/config/admin_token.new {{ garage_data_dir }}/config/admin_token
-          echo ADMIN_TOKEN_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/admin_token.new
-        fi
-      register: admin_token_write
-      changed_when: "'ADMIN_TOKEN_CHANGED' in admin_token_write.stdout"
-      no_log: true
-
-    - name: Write metrics_token
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/metrics_token.new << 'SECRET_EOF'
-        {{ garage_metrics_token }}
-        SECRET_EOF
-        chmod 600 {{ garage_data_dir }}/config/metrics_token.new
-        if [ ! -f {{ garage_data_dir }}/config/metrics_token ] \
-        || ! cmp -s {{ garage_data_dir }}/config/metrics_token.new {{ garage_data_dir }}/config/metrics_token; then
-          mv {{ garage_data_dir }}/config/metrics_token.new {{ garage_data_dir }}/config/metrics_token
-          echo METRICS_TOKEN_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/metrics_token.new
-        fi
-      register: metrics_token_write
-      changed_when: "'METRICS_TOKEN_CHANGED' in metrics_token_write.stdout"
-      no_log: true
-
-    - name: Write garage.toml
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/config/garage.toml.new << 'TOML_EOF'
-        metadata_dir = "/var/lib/garage/meta"
-        data_dir     = "/var/lib/garage/data"
-
-        # SQLite chosen over LMDB: single-node deployment has no peers to
-        # rebuild metadata from on unclean-shutdown corruption. SQLite is
-        # slower but portable and has no LMDB corruption window.
-        db_engine = "sqlite"
-
-        # 6-hour auto-snapshot of the metadata DB as a backstop against
-        # corruption; last two snapshots retained.
-        metadata_auto_snapshot_interval = "6h"
-
-        # Single-zone, single-node: only coherent replication factor.
-        replication_factor = 1
-
-        rpc_bind_addr    = "[::]:3901"
-        rpc_secret_file  = "/etc/garage/rpc_secret"
-
-        compression_level = 2
-
-        # 64 MB blocks (default 1 MB; previously 20 MB). On the 5 × 7 TB
-        # Btrfs RAID 5 volume the per-block fixed cost (SQLite lookup,
-        # dir traversal, hash verify, zstd decompress, nginx buffering)
-        # is the dominant overhead per object; larger blocks amortize
-        # it. block_size is a maximum, so 16 MiB CNPG WAL segments
-        # still upload as one block each. Only applies to NEW uploads;
-        # existing objects keep their original chunking until re-uploaded.
-        block_size = "64M"
-
-        [s3_api]
-        api_bind_addr = "[::]:3900"
-        s3_region     = "garage"
-        root_domain   = ".s3.{{ inventory_hostname }}"
-
-        [s3_web]
-        bind_addr   = "[::]:3902"
-        root_domain = ".web.{{ inventory_hostname }}"
-
-        [admin]
-        api_bind_addr      = "[::]:3903"
-        admin_token_file   = "/etc/garage/admin_token"
-        metrics_token_file = "/etc/garage/metrics_token"
-        TOML_EOF
-        if [ ! -f {{ garage_data_dir }}/config/garage.toml ] \
-        || ! cmp -s {{ garage_data_dir }}/config/garage.toml.new {{ garage_data_dir }}/config/garage.toml; then
-          mv {{ garage_data_dir }}/config/garage.toml.new {{ garage_data_dir }}/config/garage.toml
-          echo GARAGE_TOML_CHANGED
-        else
-          rm {{ garage_data_dir }}/config/garage.toml.new
-        fi
-      register: garage_toml_write
-      changed_when: "'GARAGE_TOML_CHANGED' in garage_toml_write.stdout"
-
-    - name: Write docker-compose.yml
-      ansible.builtin.raw: |
-        cat > {{ garage_data_dir }}/docker-compose.yml << 'COMPOSE_EOF'
-        services:
-          garage:
-            image: {{ garage_image }}
-            container_name: garaged
-            restart: unless-stopped
-            # HTTP services (3900/3902/3903) bind to loopback; DSM
-            # reverse proxy fronts and TLS-terminates them. RPC 3901
-            # is exposed directly — the Noise handshake is raw TCP,
-            # so DSM RP cannot front it. rpc_secret gates auth.
-            ports:
-              - "3901:3901"
-              - "127.0.0.1:3900:3900"
-              - "127.0.0.1:3902:3902"
-              - "127.0.0.1:3903:3903"
-            volumes:
-              - {{ garage_data_dir }}/config/garage.toml:/etc/garage.toml:ro
-              - {{ garage_data_dir }}/config/rpc_secret:/etc/garage/rpc_secret:ro
-              - {{ garage_data_dir }}/config/admin_token:/etc/garage/admin_token:ro
-              - {{ garage_data_dir }}/config/metrics_token:/etc/garage/metrics_token:ro
-              - {{ garage_data_dir }}/meta:/var/lib/garage/meta
-              - {{ garage_data_dir }}/data:/var/lib/garage/data
-        COMPOSE_EOF
-      changed_when: true
-
-    - name: Pull image and start Garage container
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        cd {{ garage_data_dir }} && docker compose pull && docker compose up -d
-      become: true
-      changed_when: true
-
-    # Garage does not reload config on SIGHUP and compose does not notice
-    # when a bind-mounted host file changes. Force-recreate the container
-    # when garage.toml OR any of the three secret files changed so the new
-    # config / rotated tokens are picked up.
-    - name: Force-recreate garaged when garage.toml or secrets changed
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        cd {{ garage_data_dir }} && docker compose up -d --force-recreate garage
-      become: true
-      when: garage_toml_write.changed
-            or rpc_secret_write.changed
-            or admin_token_write.changed
-            or metrics_token_write.changed
-      changed_when: true
-
-    - name: Wait for Garage to accept CLI calls
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        for i in $(seq 1 30); do
-          if docker exec garaged /garage status >/dev/null 2>&1; then
-            exit 0
-          fi
-          sleep 2
-        done
-        exit 1
-      become: true
-      changed_when: false
-
-    - name: Check whether cluster layout is already applied
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker exec garaged /garage status
-      become: true
-      register: garage_status
-      changed_when: false
-
-    - name: Apply single-node layout (idempotent — only on first run)
-      when: "'NO ROLE ASSIGNED' in garage_status.stdout"
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        NODE_ID=$(docker exec garaged /garage node id | head -n 1 | cut -d@ -f1)
-        docker exec garaged /garage layout assign "$NODE_ID" \
-          -z {{ garage_zone }} \
-          -c {{ garage_capacity }} \
-          -t {{ inventory_hostname_short }}
-        docker exec garaged /garage layout apply --version 1
-      become: true
-      changed_when: true
-
-    - name: Report Garage status
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker exec garaged /garage status
-      become: true
-      register: garage_status_final
-      changed_when: false
-
-    - name: Show Garage status
-      ansible.builtin.debug:
-        var: garage_status_final.stdout_lines
-
-    # Bootstrap buckets + keys. Idempotent: bucket_info / key_info return
-    # non-zero when missing, so we only create on first run. Access-key
-    # secrets are captured at create time and persisted to chmod-600 files
-    # under {{ garage_data_dir }}/config/keys/<key>.creds on nas-1. These
-    # files are the only post-install record of the secrets — Garage does
-    # not expose them again.
-    - name: Bootstrap Garage buckets, keys, and grants
-      ansible.builtin.shell: |
-        set -e
-        export PATH=$PATH:/usr/local/bin
-        KEYS_DIR={{ garage_data_dir }}/config/keys
-        mkdir -p "$KEYS_DIR"
-        chmod 700 "$KEYS_DIR"
-
-        bootstrap_pair() {
-          local BUCKET="$1"
-          local KEY="$2"
-          local CREDS_FILE="$KEYS_DIR/$KEY.creds"
-
-          if ! docker exec garaged /garage bucket info "$BUCKET" >/dev/null 2>&1; then
-            docker exec garaged /garage bucket create "$BUCKET"
-          fi
-
-          if ! docker exec garaged /garage key info "$KEY" >/dev/null 2>&1; then
-            CREATE_OUT=$(docker exec garaged /garage key create "$KEY")
-            KEY_ID=$(echo "$CREATE_OUT"   | awk '/^Key ID:/     {print $3}')
-            SECRET=$(echo "$CREATE_OUT"   | awk '/^Secret key:/ {print $3}')
-            umask 177
-            printf 'KEY_ID=%s\nSECRET_KEY=%s\n' "$KEY_ID" "$SECRET" > "$CREDS_FILE"
-          fi
-
-          docker exec garaged /garage bucket allow \
-            --read --write --owner \
-            "$BUCKET" --key "$KEY"
-        }
-
-        bootstrap_pair k8s-velero    velero-key
-        bootstrap_pair k8s-cnpg      cnpg-key
-        bootstrap_pair k8s-secrets   secrets-key
-        bootstrap_pair k8s-migration migration-key
-        bootstrap_pair tfstate       tfstate-key
-      become: true
-      changed_when: true
-      no_log: true
-
-    - name: Report bucket + key state (no secrets)
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker exec garaged /garage bucket list
-        echo "---"
-        docker exec garaged /garage key list
-        echo "---"
-        ls -l {{ garage_data_dir }}/config/keys/ 2>&1
-      become: true
-      register: garage_bootstrap_state
-      changed_when: false
-
-    - name: Show bucket + key state
-      ansible.builtin.debug:
-        var: garage_bootstrap_state.stdout_lines
-
-    - name: Post-install note
-      ansible.builtin.debug:
-        msg:
-          - "Buckets: k8s-velero, k8s-cnpg, k8s-secrets, k8s-migration, tfstate"
-          - "Keys:    velero-key, cnpg-key, secrets-key, migration-key, tfstate-key"
-          - "Secrets are written to {{ garage_data_dir }}/config/keys/<key>.creds on nas-1 (chmod 600, root-owned)."
-          - "SSH to nas-1 and `sudo cat` the file to retrieve KEY_ID + SECRET_KEY for Velero / CNPG / migration / tfstate config."

ansible/plays/install-pbs-client-synology.yml

@@ -1,104 +0,0 @@
----
-# Synology-specific playbook using raw module (Python 3.8 workaround)
-- name: Install PBS Client on Synology NAS
-  hosts: synology_nas
-  gather_facts: false
-
-  vars:
-    pbs_client_data_dir: "/volume1/docker/pbs-client"
-
-  tasks:
-    - name: Create PBS client directory
-      ansible.builtin.raw: |
-        mkdir -p {{ pbs_client_data_dir }}/state
-      changed_when: true
-
-    - name: Create docker-compose.yml
-      ansible.builtin.raw: |
-        cat > {{ pbs_client_data_dir }}/docker-compose.yml << 'COMPOSE_EOF'
-        services:
-          pbs-client:
-            image: proxmox/proxmox-backup-client:latest
-            container_name: pbs-client
-            restart: unless-stopped
-            hostname: {{ pbs_backup_id }}
-            env_file:
-              - .env
-            volumes:
-              - {{ pbs_client_data_dir }}/state:/root/.proxmox-backup
-        {% for path in pbs_backup_paths %}
-              - "{{ path }}:{{ path }}:ro"
-        {% endfor %}
-            mem_limit: 2g
-            command: ["sleep", "infinity"]
-        COMPOSE_EOF
-      changed_when: true
-
-    - name: Create .env file
-      ansible.builtin.raw: |
-        cat > {{ pbs_client_data_dir }}/.env << 'ENV_EOF'
-        PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
-        PBS_PASSWORD={{ pbs_password }}
-        ENV_EOF
-        chmod 600 {{ pbs_client_data_dir }}/.env
-      changed_when: true
-
-    - name: Create backup script
-      ansible.builtin.raw: |
-        cat > {{ pbs_client_data_dir }}/backup.sh << 'SCRIPT_EOF'
-        #!/bin/bash
-        set -e
-        source {{ pbs_client_data_dir }}/.env
-
-        BACKUP_SOURCES=""
-        {% for path in pbs_backup_paths %}
-        BACKUP_SOURCES="$BACKUP_SOURCES {{ path | basename }}.pxar:{{ path }}"
-        {% endfor %}
-
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting backup for {{ pbs_backup_id }}"
-        docker exec pbs-client proxmox-backup-client backup \
-            $BACKUP_SOURCES \
-            --backup-id {{ pbs_backup_id }} \
-            --repository "${PBS_REPOSITORY}"
-
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Backup completed, pruning..."
-        docker exec pbs-client proxmox-backup-client prune \
-            --repository "${PBS_REPOSITORY}" \
-            --backup-id {{ pbs_backup_id }} \
-            --keep-last {{ pbs_keep_last | default(3) }} \
-            --keep-daily {{ pbs_keep_daily | default(7) }} \
-            --keep-weekly {{ pbs_keep_weekly | default(4) }} \
-            --keep-monthly {{ pbs_keep_monthly | default(6) }}
-
-        echo "[$(date '+%Y-%m-%d %H:%M:%S')] Done"
-        SCRIPT_EOF
-        chmod 755 {{ pbs_client_data_dir }}/backup.sh
-      changed_when: true
-
-    - name: Start PBS client container
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        cd {{ pbs_client_data_dir }} && docker compose pull && docker compose up -d
-      become: true
-      changed_when: true
-
-    - name: Setup cron job for daily backup at 2 AM (root crontab)
-      ansible.builtin.shell: |
-        # Remove existing entry if present
-        crontab -l 2>/dev/null | grep -v "pbs-client/backup.sh" | crontab - 2>/dev/null || true
-        # Add new entry
-        (crontab -l 2>/dev/null; echo "0 2 * * * {{ pbs_client_data_dir }}/backup.sh >> /var/log/pbs-backup.log 2>&1") | crontab -
-      become: true
-      changed_when: true
-
-    - name: Verify container is running
-      ansible.builtin.shell: |
-        export PATH=$PATH:/usr/local/bin
-        docker ps --filter name=pbs-client --format "{{ '{{' }}.Status{{ '}}' }}"
-      become: true
-      register: container_status
-      changed_when: false
-
-    - name: Show container status
-      ansible.builtin.debug:
-        msg: "PBS client container status: {{ container_status.stdout }}"

ansible/plays/install-pbs-client.yml

@@ -1,11 +0,0 @@
----
-- name: Install PBS Client on Synology NAS
-  hosts: synology_nas
-  become: true
-  # Variables are set in:
-  # - Role defaults: ansible/roles/pbs-client/defaults/main.yml
-  # - Host overrides: ansible/inventory/host_vars/<hostname>/vars.yml
-  # - Secrets: ansible/inventory/host_vars/<hostname>/vault.yml
-
-  roles:
-    - pbs-client

ansible/roles/pbs-client/defaults/main.yml

@@ -1,32 +0,0 @@
----
-# PBS Client configuration
-pbs_client_version: "latest"
-pbs_client_data_dir: "/opt/pbs-client"
-
-# PBS Server connection (override in host_vars)
-pbs_server_host: "pbs.obg.renner.uno"
-pbs_server_port: 8007
-pbs_datastore: "backups"
-
-# Authentication (override in vault!)
-pbs_username: "backup@pbs"
-pbs_password: ""
-# pbs_fingerprint: ""  # Optional - omit for Let's Encrypt certs
-
-# Backup configuration
-pbs_backup_id: "nas-1"
-pbs_backup_paths:
-  - /volume1/docker
-  - /volume1/homes
-
-# Schedule (cron format)
-pbs_backup_schedule: "0 2 * * *"  # 2 AM daily
-
-# Retention policy (prune settings)
-pbs_keep_last: 3
-pbs_keep_daily: 7
-pbs_keep_weekly: 4
-pbs_keep_monthly: 6
-
-# Container resource limits
-pbs_memory_limit: "2g"

ansible/roles/pbs-client/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart PBS client
-  community.docker.docker_compose_v2:
-    project_src: "{{ pbs_client_data_dir }}"
-    state: restarted
-    docker_cli: "{{ docker_cli | default('/usr/local/bin/docker') }}"

ansible/roles/pbs-client/tasks/main.yml

@@ -1,23 +0,0 @@
----
-- name: Create PBS client directories
-  ansible.builtin.file:
-    path: "{{ item }}"
-    state: directory
-    mode: '0755'
-  loop:
-    - "{{ pbs_client_data_dir }}"
-    - "{{ pbs_client_data_dir }}/state"
-
-- name: Copy PBS client docker-compose file
-  ansible.builtin.template:
-    src: docker-compose.yml.j2
-    dest: "{{ pbs_client_data_dir }}/docker-compose.yml"
-    mode: '0644'
-  notify: Restart PBS client
-
-- name: Start PBS client via Docker Compose
-  community.docker.docker_compose_v2:
-    project_src: "{{ pbs_client_data_dir }}"
-    state: present
-    pull: always
-    docker_cli: "{{ docker_cli | default('/usr/local/bin/docker') }}"

ansible/roles/pbs-client/templates/.env.j2

@@ -1,6 +0,0 @@
-# PBS Client Environment Configuration
-PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
-PBS_PASSWORD={{ pbs_password }}
-{% if pbs_fingerprint | default('') != '' %}
-PBS_FINGERPRINT={{ pbs_fingerprint }}
-{% endif %}

ansible/roles/pbs-client/templates/backup.sh.j2

@@ -1,36 +0,0 @@
-#!/bin/bash
-# PBS Backup Script for {{ pbs_backup_id }}
-# Managed by Ansible - do not edit manually
-
-set -e
-
-# Load environment
-source {{ pbs_client_data_dir }}/.env
-
-# Build backup command with all paths
-BACKUP_SOURCES=""
-{% for path in pbs_backup_paths %}
-BACKUP_SOURCES="$BACKUP_SOURCES {{ path | basename }}.pxar:{{ path }}"
-{% endfor %}
-
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting backup for {{ pbs_backup_id }}"
-
-# Run backup via Docker
-docker exec pbs-client proxmox-backup-client backup \
-    $BACKUP_SOURCES \
-    --backup-id {{ pbs_backup_id }} \
-    --repository "${PBS_REPOSITORY}"
-
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Backup completed"
-
-# Prune old backups
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Pruning old backups"
-docker exec pbs-client proxmox-backup-client prune \
-    --repository "${PBS_REPOSITORY}" \
-    --backup-id {{ pbs_backup_id }} \
-    --keep-last {{ pbs_keep_last }} \
-    --keep-daily {{ pbs_keep_daily }} \
-    --keep-weekly {{ pbs_keep_weekly }} \
-    --keep-monthly {{ pbs_keep_monthly }}
-
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] Prune completed"

ansible/roles/pbs-client/templates/docker-compose.yml.j2

@@ -1,30 +0,0 @@
-services:
-  pbs-client:
-    image: aterfax/pbs-client:{{ pbs_client_version }}
-    container_name: pbs-client
-    restart: unless-stopped
-    hostname: {{ pbs_backup_id }}
-    environment:
-      - PBS_ENDPOINT={{ pbs_server_host }}:{{ pbs_server_port }}
-      - PBS_DATASTORE={{ pbs_datastore }}
-      - PBS_API_KEY_NAME={{ pbs_username }}
-      - PBS_API_KEY_SECRET={{ pbs_password }}
-      - PBS_PASSWORD={{ pbs_password }}
-      - PBS_ENCRYPTION_PASSWORD={{ pbs_encryption_password | default(pbs_password) }}
-      - PBS_REPOSITORY={{ pbs_username }}@{{ pbs_server_host }}:{{ pbs_server_port }}:{{ pbs_datastore }}
-      - PBS_BACKUP_ID={{ pbs_backup_id }}
-      - CRON_SCHEDULE={{ pbs_backup_schedule }}
-      - PBS_BACKUP_CMD_APPEND_EXTRA_OPTS=--crypt-mode=none
-      - PRUNE_KEEP_LAST={{ pbs_keep_last | default(3) }}
-      - PRUNE_KEEP_DAILY={{ pbs_keep_daily | default(7) }}
-      - PRUNE_KEEP_WEEKLY={{ pbs_keep_weekly | default(4) }}
-      - PRUNE_KEEP_MONTHLY={{ pbs_keep_monthly | default(6) }}
-      - CRON_BACKUP_ONLY=1
-    volumes:
-      - {{ pbs_client_data_dir }}/state:/root/.config/proxmox-backup
-{% for path in pbs_backup_paths %}
-      - "{{ path }}:/backups/{{ path | basename | replace(' ', '_') }}:ro"
-{% endfor %}
-    tmpfs:
-      - /tmp
-    mem_limit: {{ pbs_memory_limit }}

kubernetes/apps/file/ingress.yaml

@@ -1,31 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: file
-  namespace: apps
-  labels:
-    app.kubernetes.io/name: file
-  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-cluster
-    # Backend serves HTTPS with a self-signed cert; re-encrypt + skip
-    # verify via the shared insecure-backend ServersTransport (see
-    # apps/insecure-backend-serverstransport.yaml).
-    traefik.ingress.kubernetes.io/service.serversscheme: https
-    traefik.ingress.kubernetes.io/service.serverstransport: insecure-backend@kubernetescrd
-spec:
-  ingressClassName: traefik
-  rules:
-    - host: file.renner.uno
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: file
-                port:
-                  number: 7001
-  tls:
-    - secretName: file-tls
-      hosts:
-        - file.renner.uno

kubernetes/apps/file/kustomization.yaml

@@ -1,5 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - service.yaml
-  - ingress.yaml

kubernetes/apps/file/service.yaml

@@ -1,18 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: file
-  namespace: apps
-  labels:
-    app.kubernetes.io/name: file
-spec:
-  # Caddy reverse-proxied https://nas.renner.uno:7001. nas.renner.uno
-  # is a CNAME -> nas-1.renner.uno owned by UniFi DNS, k8s-gateway
-  # forwards srv lookups so this resolves cluster-side.
-  type: ExternalName
-  externalName: nas.renner.uno
-  ports:
-    - name: https
-      port: 7001
-      targetPort: 7001
-      protocol: TCP

kubernetes/apps/insecure-backend-serverstransport.yaml

@@ -1,20 +0,0 @@
-# Shared Traefik ServersTransport for ingresses that re-encrypt to a
-# backend serving HTTPS with a self-signed / hostname-mismatched cert:
-#
-#   apps/file       — NAS web UI on a self-signed cert
-#
-# Replaces the nginx `backend-protocol: HTTPS` + `proxy-ssl-verify: off`
-# annotation pair. Each Ingress that wants this behaviour sets
-#   traefik.ingress.kubernetes.io/service.serversscheme: https
-#   traefik.ingress.kubernetes.io/service.serverstransport: insecure-backend
-#
-# (bare name — kubernetesIngress provider resolves ServersTransport in
-# the same namespace as the Ingress; the `<ns>-<name>@kubernetescrd`
-# qualified form is for IngressRoute CRDs only).
-apiVersion: traefik.io/v1alpha1
-kind: ServersTransport
-metadata:
-  name: insecure-backend
-  namespace: apps
-spec:
-  insecureSkipVerify: true

kubernetes/apps/kustomization.yaml

@@ -1,10 +1,8 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
-  - insecure-backend-serverstransport.yaml
   - gatus/
   - external-dns/
-  - file/
   - gitea/
   - ical-proxy/
   # woodpecker/ moved to its own dedicated `woodpecker` Kustomization

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -4,7 +4,9 @@ kind: Kustomization
 # come from the kube-prometheus-stack chart. Flux applies this Kustomization
 # only after `monitoring` is Ready (chart installed → CRDs present).
 resources:
+  - prometheusrule-blackbox.yaml
   - prometheusrule-ceph.yaml
+  - prometheusrule-ci-openbao.yaml
   - prometheusrule-cilium.yaml
   - prometheusrule-disk-wear.yaml
   - prometheusrule-hetzner-backup.yaml

kubernetes/apps/monitoring-rules/probes-blackbox.yaml

@@ -49,6 +49,23 @@ spec:
         # ports outside the hairpin's 80/443 — real WAN path.
         - http://proxy-1.renner.uno:9000/health
         - http://proxy-2.renner.uno:9000/health
+        # Woodpecker. Ungated, root returns 200 with no redirect. Every
+        # scheduled job in this repo runs here — the alert-monitor cron, the
+        # nightly scans, the deploy targets — so an outage here is silent by
+        # construction: the thing that would have told you is the thing that
+        # is down.
+        - https://ci.renner.uno
+        # OpenBao. Probes the health API rather than the UI on purpose: a
+        # SEALED instance still serves the UI (307 -> /ui/ -> 200) while every
+        # ExternalSecret in the cluster fails, which is exactly the
+        # green-while-broken shape this file warns about elsewhere.
+        #
+        # standbyok=true because the endpoint answers 200 only on the ACTIVE
+        # node and 429 on a standby — the external route currently lands on a
+        # standby, so the bare path would be permanently red. With the
+        # parameter, 200 means initialized, unsealed and serving, on whichever
+        # node answers; 503 sealed and 501 uninitialized both stay red.
+        - https://openbao.renner.uno/v1/sys/health?standbyok=true
         # APP-HEALTH probes for the gated monitoring UIs, from the correct
         # in-cluster vantage (monitoring-vantage-point-rule): the gate is
         # only on the ingress, so the ClusterIP /-/healthy endpoints are

kubernetes/apps/monitoring-rules/prometheusrule-blackbox.yaml

@@ -0,0 +1,59 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: blackbox
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: blackbox.rules
+      rules:
+        # Catch-all for every blackbox-http target that has no rule of its own.
+        # Before this existed, a target could be probed, graphed and completely
+        # unalerted — which is how file.renner.uno served 502 for months.
+        #
+        # THE EXCLUSION LIST AND THE DEDICATED RULES ARE ONE INVARIANT: an
+        # instance belongs here only while a rule elsewhere covers it, and
+        # adding a dedicated rule means adding its instance here. Get it wrong
+        # in one direction and the alert fires twice; wrong in the other and it
+        # never fires at all. Currently excluded:
+        #
+        #   fsn1.your-objectstorage.com   HetznerBackupEndpointDown (critical)
+        #   netbird.renner.uno            prometheusrule-netbird-availability
+        #   proxy-1/2:9000/health         same file, relay redundancy
+        #   ci.renner.uno                 WoodpeckerDown
+        #   openbao .../sys/health        OpenBaoUnavailable
+        #
+        # Warning, not critical, on purpose: this rule cannot know what a given
+        # target means. A target whose loss deserves a critical page gets its
+        # own rule — and lands in the list above. That is what keeps the
+        # exclusion list from being arbitrary.
+        - alert: BlackboxProbeFailing
+          expr: probe_success{job="blackbox-http", instance!~"https://fsn1\\.your-objectstorage\\.com|https://netbird\\.renner\\.uno|http://proxy-[12]\\.renner\\.uno:9000/health|https://ci\\.renner\\.uno|https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
+          for: 10m
+          annotations:
+            summary: "Blackbox probe of {{ $labels.instance }} has been failing for 10 minutes"
+            description: >-
+              The endpoint is not answering as expected. For the Authentik-gated
+              hostnames this is a reachability check covering DNS, TLS, Traefik
+              and the gate outpost — it says nothing about the app behind the
+              gate, which the internal /-/healthy probes cover separately.
+          labels:
+            severity: warning
+        # The check that the checks are running. Every rule above reads
+        # probe_success; if the series stop being produced at all, none of them
+        # can fire and the silence looks exactly like health.
+        - alert: BlackboxProbesAbsent
+          expr: absent(probe_success{job="blackbox-http"})
+          for: 15m
+          annotations:
+            summary: "No blackbox-http probe results are reaching Prometheus"
+            description: >-
+              Either the blackbox exporter is down or the Probe objects stopped
+              being discovered. Every endpoint alert depends on this job, so
+              they are all blind until it returns.
+          labels:
+            severity: critical

kubernetes/apps/monitoring-rules/prometheusrule-ci-openbao.yaml

@@ -0,0 +1,47 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: ci-openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: ci-openbao.rules
+      rules:
+        - alert: WoodpeckerDown
+          # Probe target defined in probes-blackbox.yaml. Root, ungated, 200.
+          expr: probe_success{job="blackbox-http", instance="https://ci.renner.uno"} == 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Woodpecker is not answering on ci.renner.uno"
+            description: >-
+              Every scheduled job in this repo runs here: the alert-monitor
+              cron that files these issues, the nightly scans, and every deploy
+              target. Nothing else reports this outage, because the component
+              that would have reported it is the one that is down. Alertmanager
+              itself is unaffected, which is why this alert exists.
+        - alert: OpenBaoUnavailable
+          # Probes /v1/sys/health?standbyok=true — 200 on any unsealed node,
+          # 503 sealed, 501 uninitialized. The UI stays up when sealed, so a
+          # root probe would not catch this.
+          #
+          # Matched by prefix rather than the full string: the target carries a
+          # query parameter, and an exact match would break silently if the
+          # instance label ever encodes it differently.
+          expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao is sealed, uninitialized or unreachable"
+            description: >-
+              Every ExternalSecret in the cluster resolves through OpenBao.
+              Existing Secrets keep working, so nothing breaks at once — the
+              damage shows up at the next refresh interval or the next pod that
+              needs a fresh credential. Check whether it is sealed before
+              assuming a network fault.

kubernetes/apps/monitoring-rules/prometheusrule-hetzner-backup.yaml

@@ -17,11 +17,12 @@ spec:
           # drains on recovery) and Velero backups fail. Replaces the old
           # GarageDown alert now that backups live on Hetzner, not nas-1.
           #
-          # CNPG has no metric-based archiving alert yet — its pod metrics
-          # are not scraped (enablePodMonitor=false). Enabling CNPG
-          # PodMonitors for a granular "archiving falling behind" alert is
-          # a follow-up. Velero's own VeleroBackupValidationFailure /
-          # VeleroBackupStale rules already cover backup-level failures.
+          # The granular counterpart is CNPGWalArchiveBacklog in
+          # prometheusrule-cnpg-backup.yaml, reading the instance collector
+          # scraped by cnpg-podmonitors.yaml: this alert says the target is
+          # unreachable, that one says archiving is falling behind whatever the
+          # cause. Velero's VeleroBackupValidationFailure / VeleroBackupStale
+          # cover backup-level failures.
           expr: probe_success{instance="https://fsn1.your-objectstorage.com"} == 0
           for: 10m
           labels: