docs(opentofu): record the hairpin _id diff and retract the PUT 400 warning

Both of these were learned the expensive way today and existed only in commit
messages and PR descriptions, which is not where anyone looks.

providers.tf carried a warning that a UniFi PUT may require _id in the body,
which restapi does not send, and that a 400 should be answered by tainting the
object so CI recreates it. That is wrong on this controller. Tested against a
live rule with exactly the body tofu sends: HTTP 200, object returned intact,
rule byte-identical before and after, 60 rules before and after. Confirmed at
scale by apply 5675 — all 60 modified, no 400s, ci.renner.uno and
immich.renner.uno still 200.

That wrong warning had a cost. A note saying an apply might break firewall
rules is one nobody argues with, so the apply did not get run, and the
renner-uno-k8s-mariadb bucket added in b976a67b sat uncreated for three days
while the WordPress backup failed nightly with "The specified bucket does not
exist". The warning is replaced with the evidence that disproves it.

unifi_nat_hairpin.tf now records that the 60-rule `- _id` diff is permanent and
cosmetic, and lists the four attempts to remove it with the plan number that
disproved each: ignore_changes_to (5661), ignore_all_server_changes (5662),
restapi 3.x ignore_server_additions (5667), and create_returns_object (5677).

The first three cannot work in principle — `data` is a configured attribute, so
Terraform core diffs it against state before provider-side ignore logic runs.
The fourth was the better theory, that PUT responses re-polluted `data`, and it
is wrong too: the diff returned on the plan immediately after the apply. That
means the response is written into `data` on the read path, and settling it
needs the provider source rather than more guesses at its schema.

No behaviour change. Documentation only, so the next person inherits the
findings instead of the dead ends.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

.woodpecker/alert-monitor.yml

@@ -42,7 +42,7 @@ when:
 
 steps:
   - name: fetch-alerts
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/ci-manifests-test.yml

@@ -60,7 +60,7 @@ steps:
   # run that never touched them. Tracked in #461.
   - name: kustomize-build
     depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     commands:
       - |
         set -eu
@@ -115,7 +115,7 @@ steps:
   # An earlier revision of this comment also asserted that several deploy-*.yml
   # files emit "Additional property timeout is not allowed" warnings. That was
   # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
-  # still hold on v3.16.0 — review raised the likelihood that the warning was
+  # still hold on current CLIs — review raised the likelihood that the warning was
   # schema drift in a 7-minor-old CLI rather than real debt in those files, which
   # is one of the reasons this pin moved to latest stable. It cannot be confirmed
   # from a green run either, because the loop below sends lint output to /tmp/out
@@ -123,9 +123,15 @@ steps:
   # The claim is dropped rather than restated: what this step guarantees is the
   # exit-code contract above, which is what the gate actually rests on.
   # Latest stable, per the standing rule that any software uses the latest
-  # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
-  # CONSEQUENCE of that, not a separate requirement — the server is bumped by
-  # image-automation within the 3.x range, so both land on the same release.
+  # stable version. Matching the deployed woodpecker-server is a CONSEQUENCE of
+  # that, not a separate requirement — the server is bumped by image-automation
+  # within the 3.x range, so both should land on the same release.
+  #
+  # They do not land together, though: the server is a Kubernetes manifest and
+  # is bumped automatically, this pipeline file is not and cannot be (see
+  # infrastructure/image-automation/README.md). So the server moves first and
+  # this trails until someone bumps it by hand — which is what happened at
+  # v3.16.0 -> v3.17.0, caught by the update audit (#426) rather than by CI.
   #
   # It matters here because the CLI compiles the config against its own schema:
   # one behind the server can pass a config the server then rejects, or reject
@@ -152,7 +158,7 @@ steps:
   # whenever it trails the deployed server. If you bump the server, bump here.
   - name: woodpecker-lint
     depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
+    image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
     commands:
       - |
         set -eu
@@ -240,7 +246,7 @@ steps:
   # this step fails permanently on its own documentation.
   - name: no-digest-pins
     depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     commands:
       - |
         set -eu

.woodpecker/e2e-test.yml

@@ -4,7 +4,7 @@ when:
 
 steps:
   - name: e2e
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/e2e.yml

@@ -30,7 +30,7 @@ steps:
       - sh scripts/ci/test-model-selection.sh
 
   - name: e2e-test
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/issue-fix.yml

@@ -49,7 +49,7 @@ steps:
     # (the issue-pick logic needs jq, which only the ci-claude image has);
     # minting a 1h read-only token per run is cheap. Degrades to
     # snapshot-only if the ci-diagnostics ServiceAccount isn't reconciled.
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

ansible/plays/install-blackbox-exporter.yml

@@ -1,19 +0,0 @@
----
-# Deploy Blackbox Exporter to Prometheus server for endpoint probing
-# Usage: ansible-playbook -i inventory/ plays/install-blackbox-exporter.yml
-
-- name: Install Blackbox Exporter
-  hosts: prometheus.srv.renner.uno
-  become: yes
-  roles:
-    - blackbox-exporter
-
-  post_tasks:
-    - name: Verify blackbox_exporter is running
-      uri:
-        url: "http://127.0.0.1:9115/metrics"
-        status_code: 200
-      register: exporter_health
-      retries: 3
-      delay: 5
-      until: exporter_health.status == 200

ansible/plays/install-log-aggregation.yml

@@ -1,86 +0,0 @@
----
-# Setup Log Aggregation Infrastructure
-# This playbook configures the complete logging stack:
-# 1. Loki server for log storage and querying
-# 2. Alloy agents on all VMs/CTs for log collection
-# 3. Syslog receiver for remote devices (Synology NAS, network devices)
-#
-# Usage: ansible-playbook -i inventory/ plays/setup-log-aggregation.yml
-#
-# To limit to specific hosts:
-#   --limit loki.srv.renner.uno          # Only configure Loki server
-#   --limit proxmox_all_running          # Only configure log agents
-
-- name: Setup Loki Log Aggregation Server
-  hosts: proxmox_logging
-  become: yes
-  gather_facts: yes
-
-  roles:
-    - common
-    - loki-server
-
-  post_tasks:
-    - name: Verify Loki is running
-      uri:
-        url: "http://127.0.0.1:3100/ready"
-        status_code: 200
-      register: loki_health
-      retries: 5
-      delay: 10
-      until: loki_health.status == 200
-
-    - name: Display Loki status
-      debug:
-        msg: "Loki is running and healthy at http://{{ inventory_hostname }}:3100"
-
-- name: Deploy Monitoring Agent to Loki Server (with syslog receiver)
-  hosts: proxmox_logging
-  become: yes
-  gather_facts: yes
-
-  vars:
-    loki_endpoint: "http://loki.srv.renner.uno:3100"
-    prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
-    # Syslog receiver settings are defined in host_vars/loki.srv.renner.uno/vars.yml:
-    # - alloy_syslog_receiver_enabled: true
-    # - alloy_syslog_receiver_port: 1514
-    # - alloy_syslog_receiver_protocol: "udp"
-    # - alloy_syslog_receiver_format: "raw"
-    # - alloy_stability_level: "experimental"
-
-  roles:
-    - monitoring-agent
-
-  post_tasks:
-    - name: Display syslog receiver info
-      debug:
-        msg: |
-          Syslog receiver is listening on UDP port {{ alloy_syslog_receiver_port | default(1514) }}
-          Configure remote devices to send syslog to: {{ inventory_hostname }}:{{ alloy_syslog_receiver_port | default(1514) }}
-      when: alloy_syslog_receiver_enabled | default(false)
-
-- name: Deploy Monitoring Agent to All Running VMs/CTs
-  hosts: proxmox_all_running:!proxmox_logging:!proxmox_noansible
-  become: yes
-  gather_facts: yes
-  serial: 10  # Deploy to 10 hosts at a time to avoid overwhelming the network
-
-  vars:
-    loki_endpoint: "http://loki.srv.renner.uno:3100"
-    prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
-
-  roles:
-    - monitoring-agent
-
-  post_tasks:
-    - name: Verify Alloy is running
-      systemd:
-        name: alloy
-        state: started
-      register: alloy_service
-
-    - name: Display agent status
-      debug:
-        msg: "Monitoring agent deployed successfully on {{ inventory_hostname }}"
-      when: alloy_service is defined

ansible/plays/install-loki.yml

@@ -1,26 +0,0 @@
----
-# Install Loki Log Aggregation Server
-# Usage: ansible-playbook -i inventory/ plays/install-loki.yml
-
-- name: Install Loki Log Aggregation Server
-  hosts: loki.srv.renner.uno
-  become: yes
-  gather_facts: yes
-
-  roles:
-    - common
-    - loki-server
-
-  post_tasks:
-    - name: Verify Loki is running
-      uri:
-        url: "http://127.0.0.1:3100/ready"
-        status_code: 200
-      register: loki_health
-      retries: 5
-      delay: 10
-      until: loki_health.status == 200
-
-    - name: Display Loki status
-      debug:
-        msg: "Loki is running and healthy at http://{{ inventory_hostname }}:3100"

ansible/plays/install-prometheus.yml

@@ -1,43 +0,0 @@
----
-# Install Prometheus Metrics Server
-# Usage: ansible-playbook -i inventory/ plays/install-prometheus.yml
-
-- name: Install Prometheus Metrics Server
-  hosts: prometheus.srv.renner.uno
-  become: yes
-  gather_facts: yes
-
-  vars:
-    # Synology NAS SNMP targets
-    snmp_synology_targets:
-      - name: nas-1
-        target: "10.2.40.50"
-      - name: nas-2
-        target: "10.3.40.57"
-
-    # SNMPv3 credentials from vault
-    snmp_use_v3: true
-    snmp_v3_username: "{{ vault_snmp_v3_username }}"
-    snmp_v3_auth_password: "{{ vault_snmp_v3_auth_password }}"
-    snmp_v3_priv_password: "{{ vault_snmp_v3_priv_password }}"
-
-  roles:
-    - common
-    - prometheus-server
-    - snmp-exporter
-    - pve-exporter
-    - unifi-poller
-
-  post_tasks:
-    - name: Verify Prometheus is running
-      uri:
-        url: "http://127.0.0.1:9090/-/ready"
-        status_code: 200
-      register: prometheus_health
-      retries: 5
-      delay: 10
-      until: prometheus_health.status == 200
-
-    - name: Display Prometheus status
-      debug:
-        msg: "Prometheus is running and healthy at http://{{ inventory_hostname }}:9090"

ansible/plays/update.yml

@@ -16,7 +16,6 @@
 - import_playbook: install-grist.yml
 - import_playbook: install-music-assistant.yml
 - import_playbook: update-docker.yml
-- import_playbook: install-log-aggregation.yml
 - import_playbook: update-pve-rolling.yml
 
 - name: Update Summary

ansible/roles/blackbox-exporter/defaults/main.yml

@@ -1,26 +0,0 @@
----
-blackbox_exporter_version: "0.25.0"
-blackbox_exporter_port: 9115
-blackbox_exporter_user: "blackbox_exporter"
-blackbox_exporter_group: "blackbox_exporter"
-
-# Endpoints to probe
-blackbox_http_targets:
-  - name: "gitea"
-    url: "https://gitea.renner.uno"
-  - name: "immich"
-    url: "https://immich.renner.uno"
-  - name: "authentik"
-    url: "https://auth.renner.uno"
-  - name: "home-assistant"
-    url: "https://ha.renner.uno"
-  - name: "grafana"
-    url: "https://grafana.renner.uno"
-  - name: "prometheus"
-    url: "https://prometheus.renner.uno"
-
-blackbox_dns_targets: []
-
-blackbox_tcp_targets:
-  - name: "postgres-ha"
-    target: "10.2.40.135:5432"

ansible/roles/blackbox-exporter/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart blackbox_exporter
-  systemd:
-    name: blackbox_exporter
-    state: restarted
-    daemon_reload: yes

ansible/roles/blackbox-exporter/tasks/main.yml

@@ -1,80 +0,0 @@
----
-- name: Create blackbox_exporter group
-  group:
-    name: "{{ blackbox_exporter_group }}"
-    system: yes
-    state: present
-
-- name: Create blackbox_exporter user
-  user:
-    name: "{{ blackbox_exporter_user }}"
-    group: "{{ blackbox_exporter_group }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    createhome: no
-    state: present
-
-- name: Create blackbox_exporter directories
-  file:
-    path: "{{ item }}"
-    state: directory
-    owner: "{{ blackbox_exporter_user }}"
-    group: "{{ blackbox_exporter_group }}"
-    mode: '0755'
-  loop:
-    - /etc/blackbox_exporter
-    - /var/lib/blackbox_exporter
-
-- name: Download blackbox_exporter
-  get_url:
-    url: "https://github.com/prometheus/blackbox_exporter/releases/download/v{{ blackbox_exporter_version }}/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64.tar.gz"
-    dest: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.tar.gz"
-    mode: '0644'
-
-- name: Extract blackbox_exporter
-  unarchive:
-    src: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.tar.gz"
-    dest: /tmp
-    remote_src: yes
-    creates: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64"
-
-- name: Install blackbox_exporter binary
-  copy:
-    src: "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64/blackbox_exporter"
-    dest: /usr/local/bin/blackbox_exporter
-    owner: root
-    group: root
-    mode: '0755'
-    remote_src: yes
-  notify: Restart blackbox_exporter
-
-- name: Deploy blackbox_exporter configuration
-  template:
-    src: blackbox.yml.j2
-    dest: /etc/blackbox_exporter/blackbox.yml
-    owner: "{{ blackbox_exporter_user }}"
-    group: "{{ blackbox_exporter_group }}"
-    mode: '0644'
-  notify: Restart blackbox_exporter
-
-- name: Deploy blackbox_exporter systemd service
-  template:
-    src: blackbox_exporter.service.j2
-    dest: /etc/systemd/system/blackbox_exporter.service
-    mode: '0644'
-  notify: Restart blackbox_exporter
-
-- name: Enable and start blackbox_exporter
-  systemd:
-    name: blackbox_exporter
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.tar.gz"
-    - "/tmp/blackbox_exporter-{{ blackbox_exporter_version }}.linux-amd64"

ansible/roles/blackbox-exporter/templates/blackbox.yml.j2

@@ -1,60 +0,0 @@
-modules:
-  http_2xx:
-    prober: http
-    timeout: 5s
-    http:
-      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
-      valid_status_codes: []  # Defaults to 2xx
-      method: GET
-      follow_redirects: true
-      fail_if_ssl: false
-      fail_if_not_ssl: false
-      tls_config:
-        insecure_skip_verify: false
-
-  http_post_2xx:
-    prober: http
-    timeout: 5s
-    http:
-      method: POST
-      valid_status_codes: []
-
-  tcp_connect:
-    prober: tcp
-    timeout: 5s
-
-  # DNS module for external domain resolution (tests upstream/cache)
-  dns_external:
-    prober: dns
-    timeout: 5s
-    dns:
-      query_name: "google.com"
-      query_type: "A"
-      valid_rcodes:
-        - NOERROR
-
-  # DNS module for internal domain resolution (tests local resolution)
-  dns_internal:
-    prober: dns
-    timeout: 5s
-    dns:
-      query_name: "k8s-1.srv.renner.uno"
-      query_type: "A"
-      valid_rcodes:
-        - NOERROR
-
-  # DNS module for common cached domain
-  dns_cached:
-    prober: dns
-    timeout: 5s
-    dns:
-      query_name: "cloudflare.com"
-      query_type: "A"
-      valid_rcodes:
-        - NOERROR
-
-  icmp:
-    prober: icmp
-    timeout: 5s
-    icmp:
-      preferred_ip_protocol: "ip4"

ansible/roles/blackbox-exporter/templates/blackbox_exporter.service.j2

@@ -1,22 +0,0 @@
-[Unit]
-Description=Prometheus Blackbox Exporter
-Documentation=https://github.com/prometheus/blackbox_exporter
-Wants=network-online.target
-After=network-online.target
-
-[Service]
-Type=simple
-User={{ blackbox_exporter_user }}
-Group={{ blackbox_exporter_group }}
-ExecStart=/usr/local/bin/blackbox_exporter \
-    --config.file=/etc/blackbox_exporter/blackbox.yml \
-    --web.listen-address=:{{ blackbox_exporter_port }}
-Restart=on-failure
-RestartSec=5s
-
-# Allow ICMP probing (requires CAP_NET_RAW)
-AmbientCapabilities=CAP_NET_RAW
-CapabilityBoundingSet=CAP_NET_RAW
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/heating-can/tasks/main.yml

@@ -34,17 +34,42 @@
 # stopped and masked rather than merely disabled -- masked so a stray
 # `systemctl start` or a leftover dependency cannot bring them back alongside
 # the containers.
-- name: Stop and mask the native CAN services
-  ansible.builtin.systemd:
-    name: "{{ item }}"
-    state: stopped
-    enabled: false
-    masked: true
+- name: Find which native CAN units still exist
+  ansible.builtin.stat:
+    path: "/etc/systemd/system/{{ item }}.service"
   loop:
     - open3e
     - e3oncan
-  failed_when: false
+  register: heating_can_native_units
 
+# Stop and disable BEFORE anything else, and do NOT swallow the result.
+#
+# The first version did stop + disable + mask in one call with
+# failed_when: false, then deleted the unit files. Masking a unit whose file
+# still exists conflicts, so the call failed, failed_when hid it, nothing was
+# stopped -- and the next task deleted the unit files out from under two
+# running processes. systemctl is-enabled then reported "not-found" and looked
+# clean while is-active still said "active".
+#
+# The result on 2026-08-10 was two open3e instances issuing UDS requests on the
+# same CAN bus for nine minutes, fighting over the open3e-edge client ID and
+# kicking each other off the broker.
+#
+# Only units that actually exist are touched, so a fresh host does not need an
+# error swallowed to get through this.
+- name: Stop and disable the native CAN services
+  ansible.builtin.systemd:
+    name: "{{ item.item }}"
+    state: stopped
+    enabled: false
+  loop: "{{ heating_can_native_units.results }}"
+  loop_control:
+    label: "{{ item.item }}"
+  when: item.stat.exists
+
+# Masking is deliberately NOT used. It writes a symlink at the same path the
+# next task removes, so the two cancel out -- which is half of how the original
+# failed. With the unit files gone there is nothing left to start.
 - name: Remove the native unit files
   ansible.builtin.file:
     path: "/etc/systemd/system/{{ item }}.service"
@@ -54,10 +79,30 @@
     - e3oncan
   notify: Reload systemd
 
-# The venvs are the point of this change -- leaving them would leave the thing
-# that churned. Their configuration is NOT removed: devices.json, the overlay
-# and the env file are still rendered below and bind-mounted into the
-# containers.
+# The invariant that actually matters is "no native process is on the CAN bus",
+# and systemd reporting success is not the same claim. Two open3e instances
+# polling one bus is the failure this whole block exists to prevent, so it is
+# asserted directly rather than inferred from the calls above.
+- name: Check for surviving native CAN processes
+  ansible.builtin.command:
+    argv: [pgrep, -af, "edge-host/(open3e|e3oncan)"]
+  register: heating_can_survivors
+  changed_when: false
+  failed_when: false
+
+- name: Assert no native CAN process survived
+  ansible.builtin.assert:
+    that:
+      - heating_can_survivors.stdout | default('') | length == 0
+    fail_msg: >-
+      A native open3e/e3oncan process is still running after the units were
+      stopped and removed:
+      {{ heating_can_survivors.stdout_lines | default([]) | join('; ') }}.
+      Starting the container now would put two readers on the same CAN bus --
+      two open3e instances issue competing UDS requests and take over each
+      other's MQTT session. Kill the survivors, then re-run.
+    success_msg: No native CAN processes remain; safe to start the container.
+
 - name: Remove the native venvs
   ansible.builtin.file:
     path: "{{ item }}"

ansible/roles/loki-server/defaults/main.yml

@@ -1,44 +0,0 @@
----
-# Loki Server Configuration
-
-# Version and download
-loki_version: "3.4.2"
-loki_arch: "linux-amd64"
-loki_download_url: "https://github.com/grafana/loki/releases/download/v{{ loki_version }}/loki-{{ loki_arch }}.zip"
-loki_bin_path: "/usr/local/bin/loki"
-
-# Directories
-loki_data_dir: "/var/lib/loki"
-loki_config_dir: "/etc/loki"
-loki_config_file: "{{ loki_config_dir }}/loki-config.yaml"
-
-# Server configuration
-loki_http_listen_port: 3100
-loki_grpc_listen_port: 9096
-loki_http_listen_address: "0.0.0.0"
-
-# Retention settings (90 days)
-loki_retention_period: "2160h"  # 90 days
-loki_retention_delete_delay: "2h"
-
-# Storage configuration
-loki_storage_type: "filesystem"
-loki_chunks_directory: "{{ loki_data_dir }}/chunks"
-loki_rules_directory: "{{ loki_data_dir }}/rules"
-
-# Index configuration
-loki_index_prefix: "loki_index_"
-loki_index_period: "24h"
-
-# Limits
-loki_ingestion_rate_mb: 10
-loki_ingestion_burst_size_mb: 20
-loki_max_streams_per_user: 10000
-loki_max_entries_limit_per_query: 5000
-
-# Query settings
-loki_max_query_length: "720h"  # 30 days max query span
-
-# Systemd service
-loki_user: "loki"
-loki_group: "loki"

ansible/roles/loki-server/handlers/main.yml

@@ -1,10 +0,0 @@
----
-- name: Restart loki
-  systemd:
-    name: loki
-    state: restarted
-    daemon_reload: yes
-
-- name: Reload systemd
-  systemd:
-    daemon_reload: yes

ansible/roles/loki-server/tasks/main.yml

@@ -1,95 +0,0 @@
----
-# Loki Server Installation and Configuration
-
-- name: Install dependencies
-  apt:
-    name:
-      - unzip
-      - curl
-    state: present
-    update_cache: yes
-
-- name: Create loki user
-  user:
-    name: "{{ loki_user }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    create_home: no
-
-- name: Create directories
-  file:
-    path: "{{ item }}"
-    state: directory
-    owner: "{{ loki_user }}"
-    group: "{{ loki_group }}"
-    mode: '0755'
-  loop:
-    - "{{ loki_config_dir }}"
-    - "{{ loki_data_dir }}"
-    - "{{ loki_chunks_directory }}"
-    - "{{ loki_rules_directory }}"
-    - "{{ loki_data_dir }}/index"
-    - "{{ loki_data_dir }}/index_cache"
-    - "{{ loki_data_dir }}/compactor"
-
-- name: Download Loki
-  get_url:
-    url: "{{ loki_download_url }}"
-    dest: "/tmp/loki.zip"
-    mode: '0644'
-
-- name: Unzip Loki
-  unarchive:
-    src: "/tmp/loki.zip"
-    dest: "/tmp"
-    remote_src: yes
-
-- name: Install Loki binary
-  copy:
-    src: "/tmp/loki-{{ loki_arch }}"
-    dest: "{{ loki_bin_path }}"
-    mode: '0755'
-    remote_src: yes
-  notify: Restart loki
-
-- name: Create Loki configuration
-  template:
-    src: loki-config.yaml.j2
-    dest: "{{ loki_config_file }}"
-    owner: "{{ loki_user }}"
-    group: "{{ loki_group }}"
-    mode: '0644'
-  notify: Restart loki
-
-- name: Create Loki systemd service
-  template:
-    src: loki.service.j2
-    dest: /etc/systemd/system/loki.service
-    mode: '0644'
-  notify:
-    - Reload systemd
-    - Restart loki
-
-- name: Enable and start Loki service
-  systemd:
-    name: loki
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Wait for Loki to be ready
-  uri:
-    url: "http://127.0.0.1:{{ loki_http_listen_port }}/ready"
-    status_code: 200
-  register: result
-  until: result.status == 200
-  retries: 30
-  delay: 5
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "/tmp/loki.zip"
-    - "/tmp/loki-{{ loki_arch }}"

ansible/roles/loki-server/templates/loki-config.yaml.j2

@@ -1,72 +0,0 @@
-# Loki Configuration
-# Generated by Ansible
-
-auth_enabled: false
-
-server:
-  http_listen_address: {{ loki_http_listen_address }}
-  http_listen_port: {{ loki_http_listen_port }}
-  grpc_listen_port: {{ loki_grpc_listen_port }}
-  log_level: info
-
-common:
-  instance_addr: 127.0.0.1
-  path_prefix: {{ loki_data_dir }}
-  storage:
-    filesystem:
-      chunks_directory: {{ loki_chunks_directory }}
-      rules_directory: {{ loki_rules_directory }}
-  replication_factor: 1
-  ring:
-    kvstore:
-      store: inmemory
-
-query_range:
-  results_cache:
-    cache:
-      embedded_cache:
-        enabled: true
-        max_size_mb: 100
-
-schema_config:
-  configs:
-    - from: 2024-01-01
-      store: tsdb
-      object_store: filesystem
-      schema: v13
-      index:
-        prefix: {{ loki_index_prefix }}
-        period: {{ loki_index_period }}
-
-storage_config:
-  tsdb_shipper:
-    active_index_directory: {{ loki_data_dir }}/index
-    cache_location: {{ loki_data_dir }}/index_cache
-
-compactor:
-  working_directory: {{ loki_data_dir }}/compactor
-  compaction_interval: 10m
-  retention_enabled: true
-  retention_delete_delay: {{ loki_retention_delete_delay }}
-  retention_delete_worker_count: 150
-  delete_request_store: filesystem
-
-limits_config:
-  retention_period: {{ loki_retention_period }}
-  ingestion_rate_mb: {{ loki_ingestion_rate_mb }}
-  ingestion_burst_size_mb: {{ loki_ingestion_burst_size_mb }}
-  max_streams_per_user: {{ loki_max_streams_per_user }}
-  max_entries_limit_per_query: {{ loki_max_entries_limit_per_query }}
-  max_query_length: {{ loki_max_query_length }}
-  reject_old_samples: true
-  reject_old_samples_max_age: 168h  # 7 days
-
-ruler:
-  alertmanager_url: ""
-  storage:
-    type: local
-    local:
-      directory: {{ loki_rules_directory }}
-
-analytics:
-  reporting_enabled: false

ansible/roles/loki-server/templates/loki.service.j2

@@ -1,25 +0,0 @@
-[Unit]
-Description=Loki Log Aggregation System
-Documentation=https://grafana.com/docs/loki/latest/
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ loki_user }}
-Group={{ loki_group }}
-ExecStart={{ loki_bin_path }} -config.file={{ loki_config_file }}
-ExecReload=/bin/kill -HUP $MAINPID
-Restart=on-failure
-RestartSec=10s
-LimitNOFILE=65536
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-ReadWritePaths={{ loki_data_dir }}
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/prometheus-server/defaults/main.yml

@@ -1,48 +0,0 @@
----
-# Prometheus Server Configuration
-
-# Version and download
-prometheus_version: "3.2.1"
-prometheus_arch: "linux-amd64"
-prometheus_download_url: "https://github.com/prometheus/prometheus/releases/download/v{{ prometheus_version }}/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}.tar.gz"
-prometheus_bin_path: "/usr/local/bin/prometheus"
-prometheus_promtool_path: "/usr/local/bin/promtool"
-
-# Directories
-prometheus_data_dir: "/var/lib/prometheus"
-prometheus_config_dir: "/etc/prometheus"
-prometheus_config_file: "{{ prometheus_config_dir }}/prometheus.yml"
-prometheus_rules_dir: "{{ prometheus_config_dir }}/rules"
-prometheus_targets_dir: "{{ prometheus_config_dir }}/targets"
-
-# Server configuration
-prometheus_listen_address: "0.0.0.0"
-prometheus_listen_port: 9090
-prometheus_web_external_url: ""
-
-# Retention settings (90 days)
-prometheus_retention_time: "90d"
-prometheus_retention_size: "30GB"
-
-# Scrape settings
-prometheus_scrape_interval: "30s"
-prometheus_evaluation_interval: "30s"
-prometheus_scrape_timeout: "10s"
-
-# Remote write (for Alloy agents)
-prometheus_remote_write_enabled: true
-
-# Systemd service
-prometheus_user: "prometheus"
-prometheus_group: "prometheus"
-
-# Alert notification endpoints
-prometheus_alertmanager_url: ""
-
-# Email alerting (via Grafana)
-alerting_email_enabled: true
-alerting_email_to: ""
-
-# Home Assistant webhook for alerts
-alerting_ha_webhook_enabled: true
-alerting_ha_webhook_url: ""

ansible/roles/prometheus-server/handlers/main.yml

@@ -1,15 +0,0 @@
----
-- name: Restart prometheus
-  systemd:
-    name: prometheus
-    state: restarted
-    daemon_reload: yes
-
-- name: Reload prometheus
-  systemd:
-    name: prometheus
-    state: reloaded
-
-- name: Reload systemd
-  systemd:
-    daemon_reload: yes

ansible/roles/prometheus-server/tasks/main.yml

@@ -1,129 +0,0 @@
----
-# Prometheus Server Installation and Configuration
-
-- name: Install dependencies
-  apt:
-    name:
-      - curl
-    state: present
-    update_cache: yes
-
-- name: Create prometheus user
-  user:
-    name: "{{ prometheus_user }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    create_home: no
-
-- name: Create directories
-  file:
-    path: "{{ item }}"
-    state: directory
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-    mode: '0755'
-  loop:
-    - "{{ prometheus_config_dir }}"
-    - "{{ prometheus_data_dir }}"
-    - "{{ prometheus_rules_dir }}"
-    - "{{ prometheus_targets_dir }}"
-
-- name: Download Prometheus
-  get_url:
-    url: "{{ prometheus_download_url }}"
-    dest: "/tmp/prometheus.tar.gz"
-    mode: '0644'
-
-- name: Extract Prometheus
-  unarchive:
-    src: "/tmp/prometheus.tar.gz"
-    dest: "/tmp"
-    remote_src: yes
-
-- name: Install Prometheus binaries
-  copy:
-    src: "/tmp/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}/{{ item.src }}"
-    dest: "{{ item.dest }}"
-    mode: '0755'
-    remote_src: yes
-  loop:
-    - { src: "prometheus", dest: "{{ prometheus_bin_path }}" }
-    - { src: "promtool", dest: "{{ prometheus_promtool_path }}" }
-  notify: Restart prometheus
-
-- name: Check for console templates
-  stat:
-    path: "/tmp/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}/consoles"
-  register: consoles_dir
-
-- name: Copy console templates
-  copy:
-    src: "/tmp/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}/{{ item }}/"
-    dest: "{{ prometheus_config_dir }}/{{ item }}/"
-    remote_src: yes
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-  loop:
-    - consoles
-    - console_libraries
-  when: consoles_dir.stat.exists
-
-- name: Create Prometheus configuration
-  template:
-    src: prometheus.yml.j2
-    dest: "{{ prometheus_config_file }}"
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-    mode: '0644'
-    validate: "{{ prometheus_promtool_path }} check config %s"
-  notify: Restart prometheus
-
-- name: Create alert rules
-  template:
-    src: alert-rules.yml.j2
-    dest: "{{ prometheus_rules_dir }}/alert-rules.yml"
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-    mode: '0644'
-  notify: Reload prometheus
-
-- name: Create node exporter targets file
-  template:
-    src: nodes.yml.j2
-    dest: "{{ prometheus_targets_dir }}/nodes.yml"
-    owner: "{{ prometheus_user }}"
-    group: "{{ prometheus_group }}"
-    mode: '0644'
-
-- name: Create Prometheus systemd service
-  template:
-    src: prometheus.service.j2
-    dest: /etc/systemd/system/prometheus.service
-    mode: '0644'
-  notify:
-    - Reload systemd
-    - Restart prometheus
-
-- name: Enable and start Prometheus service
-  systemd:
-    name: prometheus
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Wait for Prometheus to be ready
-  uri:
-    url: "http://127.0.0.1:{{ prometheus_listen_port }}/-/ready"
-    status_code: 200
-  register: result
-  until: result.status == 200
-  retries: 30
-  delay: 5
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "/tmp/prometheus.tar.gz"
-    - "/tmp/prometheus-{{ prometheus_version }}.{{ prometheus_arch }}"

ansible/roles/prometheus-server/templates/alert-rules.yml.j2

@@ -1,100 +0,0 @@
-# Prometheus Alert Rules
-# Generated by Ansible
-
-groups:
-  - name: infrastructure
-    rules:
-      # Disk usage alerts
-      - alert: HighDiskUsage
-        expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 < 15
-        for: 5m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High disk usage on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "Disk usage is above 85% on {{ '{{' }} $labels.mountpoint {{ '}}' }} ({{ '{{' }} $value | printf \"%.1f\" {{ '}}' }}% free)"
-
-      - alert: CriticalDiskUsage
-        expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 < 5
-        for: 2m
-        labels:
-          severity: critical
-        annotations:
-          summary: "Critical disk usage on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "Disk usage is above 95% on {{ '{{' }} $labels.mountpoint {{ '}}' }} ({{ '{{' }} $value | printf \"%.1f\" {{ '}}' }}% free)"
-
-      # Memory alerts
-      - alert: HighMemoryUsage
-        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
-        for: 5m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High memory usage on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "Memory usage is above 90% ({{ '{{' }} $value | printf \"%.1f\" {{ '}}' }}%)"
-
-      # CPU alerts
-      - alert: HighCPUUsage
-        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High CPU usage on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "CPU usage is above 85% for 10 minutes ({{ '{{' }} $value | printf \"%.1f\" {{ '}}' }}%)"
-
-  - name: services
-    rules:
-      # Service down alerts
-      - alert: ServiceDown
-        expr: up == 0
-        for: 2m
-        labels:
-          severity: critical
-        annotations:
-          summary: "Service {{ '{{' }} $labels.job {{ '}}' }} down on {{ '{{' }} $labels.instance {{ '}}' }}"
-          description: "Service has been unreachable for more than 2 minutes"
-
-      # DNS VIP health
-      - alert: DNSVIPDown
-        expr: absent(up{job="dns-servers"} == 1) or count(up{job="dns-servers"} == 1) < 2
-        for: 1m
-        labels:
-          severity: critical
-        annotations:
-          summary: "DNS cluster degraded"
-          description: "Less than 2 DNS servers are healthy"
-
-  - name: security
-    rules:
-      # High auth failure rate (placeholder - needs log-based metrics from Loki)
-      - alert: HighAuthFailureRate
-        expr: rate(authentik_login_failed_total[5m]) > 5
-        for: 2m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High authentication failure rate"
-          description: "More than 5 auth failures per minute detected"
-
-  - name: loki
-    rules:
-      # Loki health
-      - alert: LokiDown
-        expr: up{job="loki"} == 0
-        for: 2m
-        labels:
-          severity: critical
-        annotations:
-          summary: "Loki log aggregation is down"
-          description: "Loki has been unreachable for more than 2 minutes"
-
-      # Loki ingestion rate
-      - alert: LokiHighIngestionRate
-        expr: rate(loki_distributor_bytes_received_total[5m]) > 10485760
-        for: 5m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High log ingestion rate in Loki"
-          description: "Log ingestion rate is above 10MB/s"

ansible/roles/prometheus-server/templates/nodes.yml.j2

@@ -1,26 +0,0 @@
-# Node Exporter targets - auto-generated by Ansible
-# Updates require re-running the prometheus playbook
-#
-# NOTE: Most hosts have Alloy agents that push metrics via remote_write.
-# This file only includes hosts that need direct scraping (standalone node_exporter).
-# Alloy-managed hosts are excluded to avoid duplicate "down" targets.
-
-# PVE cluster nodes - need direct scraping (no Alloy agent)
-- targets:
-    - 'jupiter.renner.uno:9100'
-  labels:
-    job: 'node'
-    group: 'pve_cluster'
-    hostname: 'jupiter'
-- targets:
-    - 'neptune.renner.uno:9100'
-  labels:
-    job: 'node'
-    group: 'pve_cluster'
-    hostname: 'neptune'
-- targets:
-    - 'saturn.renner.uno:9100'
-  labels:
-    job: 'node'
-    group: 'pve_cluster'
-    hostname: 'saturn'

ansible/roles/prometheus-server/templates/prometheus.service.j2

@@ -1,38 +0,0 @@
-[Unit]
-Description=Prometheus Monitoring System
-Documentation=https://prometheus.io/docs/
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ prometheus_user }}
-Group={{ prometheus_group }}
-ExecStart={{ prometheus_bin_path }} \
-  --config.file={{ prometheus_config_file }} \
-  --storage.tsdb.path={{ prometheus_data_dir }} \
-  --storage.tsdb.retention.time={{ prometheus_retention_time }} \
-  --storage.tsdb.retention.size={{ prometheus_retention_size }} \
-  --web.listen-address={{ prometheus_listen_address }}:{{ prometheus_listen_port }} \
-{% if prometheus_remote_write_enabled %}
-  --web.enable-remote-write-receiver \
-{% endif %}
-{% if prometheus_web_external_url %}
-  --web.external-url={{ prometheus_web_external_url }} \
-{% endif %}
-  --web.enable-lifecycle
-
-ExecReload=/bin/kill -HUP $MAINPID
-Restart=on-failure
-RestartSec=10s
-LimitNOFILE=65536
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-ReadWritePaths={{ prometheus_data_dir }}
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/prometheus-server/templates/prometheus.yml.j2

@@ -1,304 +0,0 @@
-# Prometheus Configuration
-# Generated by Ansible
-
-global:
-  scrape_interval: {{ prometheus_scrape_interval }}
-  evaluation_interval: {{ prometheus_evaluation_interval }}
-  scrape_timeout: {{ prometheus_scrape_timeout }}
-  external_labels:
-    cluster: homelab
-    datacenter: renner
-
-rule_files:
-  - "{{ prometheus_rules_dir }}/*.yml"
-
-{% if prometheus_alertmanager_url %}
-alerting:
-  alertmanagers:
-    - static_configs:
-        - targets:
-            - {{ prometheus_alertmanager_url }}
-{% endif %}
-
-scrape_configs:
-  # Prometheus self-monitoring
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: ['localhost:{{ prometheus_listen_port }}']
-
-  # Loki metrics
-  - job_name: 'loki'
-    static_configs:
-      - targets: ['loki.srv.renner.uno:3100']
-
-  # File-based service discovery for dynamic targets
-  - job_name: 'alloy-nodes'
-    file_sd_configs:
-      - files:
-          - '{{ prometheus_targets_dir }}/*.yml'
-          - '{{ prometheus_targets_dir }}/*.json'
-        refresh_interval: 30s
-
-{% if prometheus_remote_write_enabled %}
-# Remote write receiver for Alloy agents
-# Agents push metrics to /api/v1/write
-{% endif %}
-
-# Static targets for infrastructure components
-  # Proxmox VE API metrics (via pve-exporter)
-  - job_name: 'pve'
-    scrape_interval: 60s
-    scrape_timeout: 55s
-    static_configs:
-      - targets:
-          - 'jupiter.renner.uno'
-          - 'neptune.renner.uno'
-          - 'saturn.renner.uno'
-    metrics_path: /pve
-    params:
-      module: [default]
-      cluster: ['1']
-      node: ['1']
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9221  # pve-exporter address
-      - target_label: group
-        replacement: proxmox
-      - target_label: type
-        replacement: hypervisor
-
-  # RustFS (S3-compatible storage)
-  # NOTE: RustFS uses OpenTelemetry push-model via --obs-endpoint, not Prometheus pull.
-  # Unlike MinIO, it doesn't expose /minio/v2/metrics/cluster endpoint.
-  # Metrics collection would require setting up RUSTFS_OBS_ENDPOINT to push to an OTEL collector.
-  # For now, node-level metrics are available via Alloy agents on the RustFS hosts.
-
-  # Home Assistant metrics
-  - job_name: 'homeassistant'
-    metrics_path: /api/prometheus
-    bearer_token: '{{ homeassistant_prometheus_token }}'
-    scheme: http
-    static_configs:
-      - targets: ['homeassistant.iot.renner.uno:8123']
-        labels:
-          group: 'smarthome'
-          type: 'homeassistant'
-
-  # UniFi Poller metrics
-  - job_name: 'unifi'
-    static_configs:
-      - targets: ['localhost:9130']
-        labels:
-          group: 'network'
-          type: 'unifi'
-
-  # Caddy reverse proxy metrics (HA cluster)
-  - job_name: 'caddy'
-    static_configs:
-      - targets:
-          - 'caddy.dmz.renner.uno:2019'
-          - 'caddy2.dmz.renner.uno:2019'
-        labels:
-          group: 'ingress'
-          type: 'reverse-proxy'
-
-  # PostgreSQL metrics (via postgres_exporter)
-  - job_name: 'postgresql'
-    static_configs:
-      - targets: ['localhost:9187']
-        labels:
-          group: 'database'
-          type: 'postgresql'
-
-  # Authentik identity provider metrics
-  - job_name: 'authentik'
-    static_configs:
-      - targets:
-          - 'authentik-app-1.srv.renner.uno:9300'
-          - 'authentik-app-2.srv.renner.uno:9300'
-        labels:
-          group: 'identity'
-          type: 'authentik'
-
-  # Node Exporter - system metrics from all hosts
-  - job_name: 'node'
-    file_sd_configs:
-      - files:
-          - '{{ prometheus_targets_dir }}/nodes.yml'
-        refresh_interval: 60s
-
-  # Chrony Exporter - NTP metrics from PVE cluster
-  # One static_config per host so each carries its own hostname label. The
-  # previous form listed all three targets together and recovered the hostname
-  # in relabel_configs by regex-matching __address__ against the node's IP --
-  # which silently coupled the labels to the addressing. Changing a target
-  # would have left the regex unmatched and the hostname label simply absent,
-  # with the scrape still succeeding.
-  - job_name: 'chrony'
-    static_configs:
-      - targets: ['jupiter.renner.uno:9123']
-        labels:
-          group: 'ntp'
-          type: 'time-server'
-          hostname: 'jupiter'
-      - targets: ['neptune.renner.uno:9123']
-        labels:
-          group: 'ntp'
-          type: 'time-server'
-          hostname: 'neptune'
-      - targets: ['saturn.renner.uno:9123']
-        labels:
-          group: 'ntp'
-          type: 'time-server'
-          hostname: 'saturn'
-
-  # Immich photo management metrics
-  - job_name: 'immich'
-    static_configs:
-      - targets: ['immich.srv.renner.uno:8081']
-        labels:
-          group: 'media'
-          type: 'photo-management'
-
-  # Blackbox Exporter - HTTP endpoint probing
-  - job_name: 'blackbox-http'
-    metrics_path: /probe
-    params:
-      module: [http_2xx]
-    static_configs:
-      - targets:
-          - https://gitea.renner.uno
-          - https://immich.renner.uno
-          - https://auth.renner.uno
-          - https://ha.renner.uno
-          - https://grafana.renner.uno
-          - https://prometheus.renner.uno
-        labels:
-          group: 'probe'
-          type: 'http'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-  # Blackbox Exporter - TCP probing
-  - job_name: 'blackbox-tcp'
-    metrics_path: /probe
-    params:
-      module: [tcp_connect]
-    static_configs:
-      - targets:
-          - 10.2.40.135:5432  # PostgreSQL HA VIP
-        labels:
-          group: 'probe'
-          type: 'tcp'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-  # Blackbox Exporter - DNS probing (external domains like google.com)
-  - job_name: 'blackbox-dns-external'
-    metrics_path: /probe
-    params:
-      module: [dns_external]
-    static_configs:
-      - targets:
-          - 10.2.40.53  # Proxmox DNS VIP
-          - 10.2.40.1   # UDM Gateway DNS
-        labels:
-          group: 'probe'
-          type: 'dns'
-          query_type: 'external'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-  # Blackbox Exporter - DNS probing (internal domains like k8s-1.srv.renner.uno)
-  - job_name: 'blackbox-dns-internal'
-    metrics_path: /probe
-    params:
-      module: [dns_internal]
-    static_configs:
-      - targets:
-          - 10.2.40.53  # Proxmox DNS VIP
-          - 10.2.40.1   # UDM Gateway DNS
-        labels:
-          group: 'probe'
-          type: 'dns'
-          query_type: 'internal'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-  # Blackbox Exporter - DNS probing (cached domains)
-  - job_name: 'blackbox-dns-cached'
-    metrics_path: /probe
-    params:
-      module: [dns_cached]
-    static_configs:
-      - targets:
-          - 10.2.40.53  # Proxmox DNS VIP
-          - 10.2.40.1   # UDM Gateway DNS
-        labels:
-          group: 'probe'
-          type: 'dns'
-          query_type: 'cached'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__param_target]
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9115
-
-{% if snmp_synology_targets is defined and snmp_synology_targets | length > 0 %}
-  # Synology NAS SNMP metrics
-  # snmp_exporter v0.27+ requires both synology and ucd_la_table modules
-  - job_name: 'synology'
-    scrape_interval: 60s
-    scrape_timeout: 55s
-    metrics_path: /snmp
-    params:
-      module: [synology, ucd_la_table, ucd_memory, if_mib]
-{% if snmp_use_v3 | default(false) %}
-      auth: [synology_v3]
-{% else %}
-      auth: [synology_v2]
-{% endif %}
-    static_configs:
-      - targets:
-{% for target in snmp_synology_targets %}
-          - '{{ target.target }}'  # {{ target.name }}
-{% endfor %}
-        labels:
-          group: 'storage'
-          type: 'nas'
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_target
-      - source_labels: [__address__]
-        regex: '([^:]+)(:\d+)?'
-        replacement: '${1}'
-        target_label: instance
-      - target_label: __address__
-        replacement: localhost:9116  # snmp_exporter address
-{% endif %}

ansible/roles/promtail/defaults/main.yml

@@ -1,36 +0,0 @@
----
-# Promtail Configuration
-
-# Version and download
-promtail_version: "3.4.2"
-promtail_arch: "linux-amd64"
-promtail_download_url: "https://github.com/grafana/loki/releases/download/v{{ promtail_version }}/promtail-{{ promtail_arch }}.zip"
-promtail_bin_path: "/usr/local/bin/promtail"
-
-# Directories
-promtail_config_dir: "/etc/promtail"
-promtail_config_file: "{{ promtail_config_dir }}/promtail-config.yaml"
-promtail_positions_file: "/var/lib/promtail/positions.yaml"
-
-# Server configuration
-promtail_http_listen_port: 9080
-promtail_http_listen_address: "0.0.0.0"
-
-# Loki server
-promtail_loki_url: "http://loki.srv.renner.uno:3100/loki/api/v1/push"
-
-# Log scrape configurations
-# Override in playbook or host_vars for specific services
-promtail_scrape_configs:
-  - job_name: system
-    static_configs:
-      - targets:
-          - localhost
-        labels:
-          job: syslog
-          host: "{{ inventory_hostname }}"
-          __path__: /var/log/syslog
-
-# Systemd service
-promtail_user: "promtail"
-promtail_group: "promtail"

ansible/roles/promtail/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart promtail
-  systemd:
-    name: promtail
-    state: restarted
-    daemon_reload: yes

ansible/roles/promtail/tasks/main.yml

@@ -1,112 +0,0 @@
----
-- name: Install dependencies
-  apt:
-    name:
-      - unzip
-      - curl
-    state: present
-    update_cache: yes
-
-- name: Create promtail group
-  group:
-    name: "{{ promtail_group }}"
-    system: yes
-    state: present
-
-- name: Create promtail user
-  user:
-    name: "{{ promtail_user }}"
-    group: "{{ promtail_group }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    home: /var/lib/promtail
-    create_home: yes
-    state: present
-
-- name: Add promtail user to systemd-journal group
-  user:
-    name: "{{ promtail_user }}"
-    groups: systemd-journal
-    append: yes
-
-- name: Create config directory
-  file:
-    path: "{{ promtail_config_dir }}"
-    state: directory
-    owner: "{{ promtail_user }}"
-    group: "{{ promtail_group }}"
-    mode: '0755'
-
-- name: Create positions directory
-  file:
-    path: "{{ promtail_positions_file | dirname }}"
-    state: directory
-    owner: "{{ promtail_user }}"
-    group: "{{ promtail_group }}"
-    mode: '0755'
-
-- name: Check if promtail binary exists
-  stat:
-    path: "{{ promtail_bin_path }}"
-  register: promtail_binary
-
-- name: Get installed promtail version
-  command: "{{ promtail_bin_path }} --version"
-  register: installed_version
-  changed_when: false
-  failed_when: false
-  when: promtail_binary.stat.exists
-
-- name: Download promtail
-  get_url:
-    url: "{{ promtail_download_url }}"
-    dest: /tmp/promtail.zip
-    mode: '0644'
-  when: not promtail_binary.stat.exists or (installed_version.stdout is defined and promtail_version not in installed_version.stdout)
-
-- name: Extract promtail binary
-  unarchive:
-    src: /tmp/promtail.zip
-    dest: /tmp/
-    remote_src: yes
-  when: not promtail_binary.stat.exists or (installed_version.stdout is defined and promtail_version not in installed_version.stdout)
-
-- name: Install promtail binary
-  copy:
-    src: "/tmp/promtail-{{ promtail_arch }}"
-    dest: "{{ promtail_bin_path }}"
-    remote_src: yes
-    mode: '0755'
-  when: not promtail_binary.stat.exists or (installed_version.stdout is defined and promtail_version not in installed_version.stdout)
-  notify: Restart promtail
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - /tmp/promtail.zip
-    - "/tmp/promtail-{{ promtail_arch }}"
-
-- name: Configure promtail
-  template:
-    src: promtail-config.yaml.j2
-    dest: "{{ promtail_config_file }}"
-    owner: "{{ promtail_user }}"
-    group: "{{ promtail_group }}"
-    mode: '0644'
-  notify: Restart promtail
-
-- name: Create systemd service
-  template:
-    src: promtail.service.j2
-    dest: /etc/systemd/system/promtail.service
-    mode: '0644'
-  notify: Restart promtail
-
-- name: Enable and start promtail
-  systemd:
-    name: promtail
-    state: started
-    enabled: yes
-    daemon_reload: yes

ansible/roles/promtail/templates/promtail-config.yaml.j2

@@ -1,79 +0,0 @@
-server:
-  http_listen_port: {{ promtail_http_listen_port }}
-  http_listen_address: {{ promtail_http_listen_address }}
-  grpc_listen_port: 0
-
-positions:
-  filename: {{ promtail_positions_file }}
-
-clients:
-  - url: {{ promtail_loki_url }}
-    tenant_id: default
-
-scrape_configs:
-{% for config in promtail_scrape_configs %}
-  - job_name: {{ config.job_name }}
-{% if config.journal is defined %}
-    journal:
-      max_age: {{ config.journal.max_age | default('12h') }}
-      labels:
-        job: {{ config.job_name }}
-        host: {{ inventory_hostname }}
-{% if config.journal.matches is defined %}
-      matches: {{ config.journal.matches }}
-{% endif %}
-{% if config.relabel_configs is defined %}
-    relabel_configs:
-{% for relabel in config.relabel_configs %}
-      - source_labels: {{ relabel.source_labels | to_json }}
-        target_label: {{ relabel.target_label }}
-{% if relabel.regex is defined %}
-        regex: '{{ relabel.regex }}'
-{% endif %}
-{% if relabel.replacement is defined %}
-        replacement: '{{ relabel.replacement }}'
-{% endif %}
-{% endfor %}
-{% endif %}
-{% endif %}
-{% if config.static_configs is defined %}
-    static_configs:
-{% for static in config.static_configs %}
-      - targets:
-{% for target in static.targets %}
-          - {{ target }}
-{% endfor %}
-        labels:
-{% for key, value in static.labels.items() %}
-          {{ key }}: {{ value }}
-{% endfor %}
-{% endfor %}
-{% if config.pipeline_stages is defined %}
-    pipeline_stages:
-{% for stage in config.pipeline_stages %}
-{% if stage.multiline is defined %}
-      - multiline:
-          firstline: '{{ stage.multiline.firstline }}'
-{% if stage.multiline.max_wait_time is defined %}
-          max_wait_time: {{ stage.multiline.max_wait_time }}
-{% endif %}
-{% endif %}
-{% if stage.regex is defined %}
-      - regex:
-          expression: '{{ stage.regex.expression }}'
-{% endif %}
-{% if stage.labels is defined %}
-      - labels:
-{% for label in stage.labels %}
-          {{ label }}:
-{% endfor %}
-{% endif %}
-{% if stage.timestamp is defined %}
-      - timestamp:
-          source: {{ stage.timestamp.source }}
-          format: '{{ stage.timestamp.format }}'
-{% endif %}
-{% endfor %}
-{% endif %}
-{% endif %}
-{% endfor %}

ansible/roles/promtail/templates/promtail.service.j2

@@ -1,24 +0,0 @@
-[Unit]
-Description=Promtail Log Collector
-Documentation=https://grafana.com/docs/loki/latest/clients/promtail/
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ promtail_user }}
-Group={{ promtail_group }}
-ExecStart={{ promtail_bin_path }} -config.file={{ promtail_config_file }}
-Restart=always
-RestartSec=5
-TimeoutStopSec=30
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-ReadWritePaths=/var/lib/promtail
-CapabilityBoundingSet=CAP_DAC_READ_SEARCH
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/pve-exporter/defaults/main.yml

@@ -1,19 +0,0 @@
----
-# PVE Exporter defaults
-
-# Installation
-pve_exporter_version: "3.4.5"
-pve_exporter_install_dir: /opt/pve-exporter
-pve_exporter_config_dir: /etc/pve-exporter
-pve_exporter_venv_path: "{{ pve_exporter_install_dir }}/venv"
-
-# Service
-pve_exporter_user: pve-exporter
-pve_exporter_group: pve-exporter
-pve_exporter_port: 9221
-
-# Proxmox connection (override with vault variables)
-pve_exporter_pve_user: "{{ pve_user | default('root@pam') }}"
-pve_exporter_pve_token_name: "{{ pve_token_id | default('ansible') }}"
-pve_exporter_pve_token_value: "{{ pve_token_secret }}"
-pve_exporter_pve_verify_ssl: false

ansible/roles/pve-exporter/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart pve-exporter
-  systemd:
-    name: pve-exporter
-    state: restarted
-    daemon_reload: yes

ansible/roles/pve-exporter/tasks/main.yml

@@ -1,78 +0,0 @@
----
-# PVE Exporter installation and configuration
-
-- name: Install Python dependencies
-  apt:
-    name:
-      - python3
-      - python3-pip
-      - python3-venv
-    state: present
-    update_cache: yes
-
-- name: Create pve-exporter system user
-  user:
-    name: "{{ pve_exporter_user }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    create_home: no
-
-- name: Create installation directory
-  file:
-    path: "{{ pve_exporter_install_dir }}"
-    state: directory
-    owner: "{{ pve_exporter_user }}"
-    group: "{{ pve_exporter_group }}"
-    mode: '0755'
-
-- name: Create configuration directory
-  file:
-    path: "{{ pve_exporter_config_dir }}"
-    state: directory
-    owner: "{{ pve_exporter_user }}"
-    group: "{{ pve_exporter_group }}"
-    mode: '0750'
-
-- name: Create Python virtual environment
-  command: python3 -m venv {{ pve_exporter_venv_path }}
-  args:
-    creates: "{{ pve_exporter_venv_path }}/bin/python"
-
-- name: Set venv ownership
-  file:
-    path: "{{ pve_exporter_venv_path }}"
-    state: directory
-    owner: "{{ pve_exporter_user }}"
-    group: "{{ pve_exporter_group }}"
-    recurse: yes
-
-- name: Install prometheus-pve-exporter
-  pip:
-    name: prometheus-pve-exporter
-    virtualenv: "{{ pve_exporter_venv_path }}"
-    state: present
-
-- name: Create configuration file
-  template:
-    src: pve.yml.j2
-    dest: "{{ pve_exporter_config_dir }}/pve.yml"
-    owner: "{{ pve_exporter_user }}"
-    group: "{{ pve_exporter_group }}"
-    mode: '0640'
-  notify: Restart pve-exporter
-
-- name: Create systemd service
-  template:
-    src: pve-exporter.service.j2
-    dest: /etc/systemd/system/pve-exporter.service
-    owner: root
-    group: root
-    mode: '0644'
-  notify: Restart pve-exporter
-
-- name: Enable and start pve-exporter service
-  systemd:
-    name: pve-exporter
-    state: started
-    enabled: yes
-    daemon_reload: yes

ansible/roles/pve-exporter/templates/pve-exporter.service.j2

@@ -1,22 +0,0 @@
-[Unit]
-Description=Prometheus PVE Exporter
-Documentation=https://github.com/prometheus-pve/prometheus-pve-exporter
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ pve_exporter_user }}
-Group={{ pve_exporter_group }}
-ExecStart={{ pve_exporter_venv_path }}/bin/pve_exporter \
-  --config.file {{ pve_exporter_config_dir }}/pve.yml \
-  --web.listen-address 0.0.0.0:{{ pve_exporter_port }}
-Restart=on-failure
-RestartSec=10s
-
-# Security hardening
-NoNewPrivileges=yes
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/pve-exporter/templates/pve.yml.j2

@@ -1,8 +0,0 @@
-# PVE Exporter Configuration
-# Generated by Ansible
-
-default:
-  user: "{{ pve_exporter_pve_user }}"
-  token_name: "{{ pve_exporter_pve_token_name }}"
-  token_value: "{{ pve_exporter_pve_token_value }}"
-  verify_ssl: {{ pve_exporter_pve_verify_ssl | lower }}

ansible/roles/snmp-exporter/defaults/main.yml

@@ -1,33 +0,0 @@
----
-# SNMP Exporter defaults
-
-snmp_exporter_version: "0.30.1"
-snmp_exporter_listen_port: 9116
-snmp_exporter_user: "snmp_exporter"
-snmp_exporter_group: "snmp_exporter"
-
-snmp_exporter_install_dir: "/opt/snmp_exporter"
-snmp_exporter_config_dir: "/etc/snmp_exporter"
-
-# Synology NAS SNMP targets
-# Configure in host_vars or group_vars
-snmp_synology_targets: []
-# Example:
-# snmp_synology_targets:
-#   - name: nas-1
-#     target: 10.2.40.50
-#   - name: nas-2
-#     target: 10.2.40.51
-
-# SNMPv3 authentication (recommended for security)
-# These should be set in vault
-snmp_v3_username: "prometheus"
-snmp_v3_auth_protocol: "SHA"  # MD5 or SHA
-snmp_v3_auth_password: ""     # Set in vault
-snmp_v3_priv_protocol: "AES"  # DES or AES
-snmp_v3_priv_password: ""     # Set in vault
-snmp_v3_security_level: "authPriv"  # noAuthNoPriv, authNoPriv, authPriv
-
-# SNMPv2c fallback (less secure, but simpler)
-snmp_v2_community: "public"
-snmp_use_v3: true

ansible/roles/snmp-exporter/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart snmp_exporter
-  systemd:
-    name: snmp_exporter
-    state: restarted
-    daemon_reload: yes

ansible/roles/snmp-exporter/tasks/main.yml

@@ -1,114 +0,0 @@
----
-# SNMP Exporter installation and configuration
-
-- name: Create snmp_exporter group
-  group:
-    name: "{{ snmp_exporter_group }}"
-    system: yes
-    state: present
-
-- name: Create snmp_exporter user
-  user:
-    name: "{{ snmp_exporter_user }}"
-    group: "{{ snmp_exporter_group }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    home: "{{ snmp_exporter_install_dir }}"
-    create_home: no
-
-- name: Create directories
-  file:
-    path: "{{ item }}"
-    state: directory
-    owner: "{{ snmp_exporter_user }}"
-    group: "{{ snmp_exporter_group }}"
-    mode: '0755'
-  loop:
-    - "{{ snmp_exporter_install_dir }}"
-    - "{{ snmp_exporter_config_dir }}"
-
-- name: Check if snmp_exporter is already installed
-  stat:
-    path: "{{ snmp_exporter_install_dir }}/snmp_exporter"
-  register: snmp_exporter_binary
-
-- name: Get installed snmp_exporter version
-  command: "{{ snmp_exporter_install_dir }}/snmp_exporter --version"
-  register: installed_version
-  changed_when: false
-  failed_when: false
-  when: snmp_exporter_binary.stat.exists
-
-- name: Download snmp_exporter
-  get_url:
-    url: "https://github.com/prometheus/snmp_exporter/releases/download/v{{ snmp_exporter_version }}/snmp_exporter-{{ snmp_exporter_version }}.linux-amd64.tar.gz"
-    dest: "/tmp/snmp_exporter.tar.gz"
-    mode: '0644'
-  when: not snmp_exporter_binary.stat.exists or (installed_version.stdout is defined and snmp_exporter_version not in installed_version.stdout)
-
-- name: Extract snmp_exporter
-  unarchive:
-    src: "/tmp/snmp_exporter.tar.gz"
-    dest: "/tmp/"
-    remote_src: yes
-  when: not snmp_exporter_binary.stat.exists or (installed_version.stdout is defined and snmp_exporter_version not in installed_version.stdout)
-
-- name: Install snmp_exporter binary
-  copy:
-    src: "/tmp/snmp_exporter-{{ snmp_exporter_version }}.linux-amd64/snmp_exporter"
-    dest: "{{ snmp_exporter_install_dir }}/snmp_exporter"
-    remote_src: yes
-    owner: "{{ snmp_exporter_user }}"
-    group: "{{ snmp_exporter_group }}"
-    mode: '0755'
-  when: not snmp_exporter_binary.stat.exists or (installed_version.stdout is defined and snmp_exporter_version not in installed_version.stdout)
-  notify: Restart snmp_exporter
-
-- name: Install default snmp.yml from release
-  copy:
-    src: "/tmp/snmp_exporter-{{ snmp_exporter_version }}.linux-amd64/snmp.yml"
-    dest: "{{ snmp_exporter_config_dir }}/snmp.yml"
-    remote_src: yes
-    owner: "{{ snmp_exporter_user }}"
-    group: "{{ snmp_exporter_group }}"
-    mode: '0644'
-  when: not snmp_exporter_binary.stat.exists or (installed_version.stdout is defined and snmp_exporter_version not in installed_version.stdout)
-  notify: Restart snmp_exporter
-
-- name: Add synology_v3 auth entry
-  lineinfile:
-    path: "{{ snmp_exporter_config_dir }}/snmp.yml"
-    insertafter: "^auths:"
-    line: "  synology_v3: {version: 3, security_level: {{ snmp_v3_security_level }}, username: {{ snmp_v3_username }}, password: {{ snmp_v3_auth_password }}, auth_protocol: {{ snmp_v3_auth_protocol }}, priv_protocol: {{ snmp_v3_priv_protocol }}, priv_password: {{ snmp_v3_priv_password }}}"
-  notify: Restart snmp_exporter
-
-- name: Add synology_v2 auth entry
-  lineinfile:
-    path: "{{ snmp_exporter_config_dir }}/snmp.yml"
-    insertafter: "^auths:"
-    line: "  synology_v2: {version: 2, community: {{ snmp_v2_community }}}"
-  notify: Restart snmp_exporter
-
-- name: Create snmp_exporter systemd service
-  template:
-    src: snmp_exporter.service.j2
-    dest: /etc/systemd/system/snmp_exporter.service
-    owner: root
-    group: root
-    mode: '0644'
-  notify: Restart snmp_exporter
-
-- name: Enable and start snmp_exporter service
-  systemd:
-    name: snmp_exporter
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "/tmp/snmp_exporter.tar.gz"
-    - "/tmp/snmp_exporter-{{ snmp_exporter_version }}.linux-amd64"

ansible/roles/snmp-exporter/templates/snmp-auth.yml.j2

@@ -1,18 +0,0 @@
-# SNMPv3 Authentication for Synology NAS
-# Appended to default snmp.yml by Ansible
-
-auths:
-{% if snmp_use_v3 | default(true) %}
-  synology_v3:
-    security_level: {{ snmp_v3_security_level }}
-    username: {{ snmp_v3_username }}
-    password: {{ snmp_v3_auth_password }}
-    auth_protocol: {{ snmp_v3_auth_protocol }}
-{% if snmp_v3_security_level == 'authPriv' %}
-    priv_protocol: {{ snmp_v3_priv_protocol }}
-    priv_password: {{ snmp_v3_priv_password }}
-{% endif %}
-{% else %}
-  synology_v2:
-    community: {{ snmp_v2_community }}
-{% endif %}

ansible/roles/snmp-exporter/templates/snmp_exporter.service.j2

@@ -1,26 +0,0 @@
-[Unit]
-Description=Prometheus SNMP Exporter
-Documentation=https://github.com/prometheus/snmp_exporter
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ snmp_exporter_user }}
-Group={{ snmp_exporter_group }}
-ExecStart={{ snmp_exporter_install_dir }}/snmp_exporter \
-  --config.file={{ snmp_exporter_config_dir }}/snmp.yml \
-  --web.listen-address=:{{ snmp_exporter_listen_port }}
-
-Restart=on-failure
-RestartSec=5s
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-PrivateTmp=yes
-ReadOnlyPaths=/
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/unifi-poller/defaults/main.yml

@@ -1,34 +0,0 @@
----
-# UniFi Poller defaults
-
-# Version to install
-unifi_poller_version: "2.11.2"
-
-# Installation paths
-unifi_poller_bin_path: /usr/local/bin/unpoller
-unifi_poller_config_dir: /etc/unpoller
-unifi_poller_config_file: "{{ unifi_poller_config_dir }}/up.conf"
-
-# Service user
-unifi_poller_user: unpoller
-unifi_poller_group: unpoller
-
-# UniFi Controller settings (override in vars/vault)
-unifi_controller_url: "https://192.168.1.1"
-unifi_controller_user: ""
-unifi_controller_pass: ""
-unifi_controller_verify_ssl: false
-unifi_controller_site: "default"
-
-# Prometheus exporter settings
-unifi_poller_prometheus_port: 9130
-unifi_poller_prometheus_path: /metrics
-
-# Polling intervals
-unifi_poller_interval: 30s
-unifi_poller_save_sites: true
-unifi_poller_save_ids: false
-unifi_poller_save_events: false
-unifi_poller_save_alarms: false
-unifi_poller_save_anomalies: false
-unifi_poller_save_dpi: true

ansible/roles/unifi-poller/handlers/main.yml

@@ -1,6 +0,0 @@
----
-- name: Restart unpoller
-  systemd:
-    name: unpoller
-    state: restarted
-    daemon_reload: yes

ansible/roles/unifi-poller/tasks/main.yml

@@ -1,88 +0,0 @@
----
-# UniFi Poller installation and configuration
-
-- name: Create unpoller system user
-  user:
-    name: "{{ unifi_poller_user }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    create_home: no
-
-- name: Create configuration directory
-  file:
-    path: "{{ unifi_poller_config_dir }}"
-    state: directory
-    owner: "{{ unifi_poller_user }}"
-    group: "{{ unifi_poller_group }}"
-    mode: '0750'
-
-- name: Check if unpoller is already installed
-  stat:
-    path: "{{ unifi_poller_bin_path }}"
-  register: unpoller_binary
-
-- name: Get installed version
-  command: "{{ unifi_poller_bin_path }} --version"
-  register: installed_version
-  changed_when: false
-  failed_when: false
-  when: unpoller_binary.stat.exists
-
-- name: Download UniFi Poller
-  get_url:
-    url: "https://github.com/unpoller/unpoller/releases/download/v{{ unifi_poller_version }}/unpoller_{{ unifi_poller_version }}_linux_amd64.tar.gz"
-    dest: /tmp/unpoller.tar.gz
-    mode: '0644'
-  when: not unpoller_binary.stat.exists or (installed_version.stdout is defined and unifi_poller_version not in installed_version.stdout)
-
-- name: Extract UniFi Poller
-  unarchive:
-    src: /tmp/unpoller.tar.gz
-    dest: /tmp/
-    remote_src: yes
-  when: not unpoller_binary.stat.exists or (installed_version.stdout is defined and unifi_poller_version not in installed_version.stdout)
-
-- name: Install UniFi Poller binary
-  copy:
-    src: /tmp/unpoller
-    dest: "{{ unifi_poller_bin_path }}"
-    remote_src: yes
-    owner: root
-    group: root
-    mode: '0755'
-  when: not unpoller_binary.stat.exists or (installed_version.stdout is defined and unifi_poller_version not in installed_version.stdout)
-  notify: Restart unpoller
-
-- name: Create configuration file
-  template:
-    src: up.conf.j2
-    dest: "{{ unifi_poller_config_file }}"
-    owner: "{{ unifi_poller_user }}"
-    group: "{{ unifi_poller_group }}"
-    mode: '0640'
-  notify: Restart unpoller
-
-- name: Create systemd service
-  template:
-    src: unpoller.service.j2
-    dest: /etc/systemd/system/unpoller.service
-    owner: root
-    group: root
-    mode: '0644'
-  notify: Restart unpoller
-
-- name: Enable and start unpoller service
-  systemd:
-    name: unpoller
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
-- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - /tmp/unpoller.tar.gz
-    - /tmp/unpoller
-  changed_when: false

ansible/roles/unifi-poller/templates/unpoller.service.j2

@@ -1,24 +0,0 @@
-[Unit]
-Description=UniFi Poller - Prometheus Exporter for UniFi Network
-Documentation=https://github.com/unpoller/unpoller
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ unifi_poller_user }}
-Group={{ unifi_poller_group }}
-ExecStart={{ unifi_poller_bin_path }} --config {{ unifi_poller_config_file }}
-Restart=on-failure
-RestartSec=10s
-
-# Security hardening
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-ReadOnlyPaths=/
-ReadWritePaths={{ unifi_poller_config_dir }}
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target

ansible/roles/unifi-poller/templates/up.conf.j2

@@ -1,42 +0,0 @@
-# UniFi Poller Configuration
-# Generated by Ansible
-
-[poller]
-  # Turns on debug messages
-  debug = false
-  # Turns off messages
-  quiet = false
-
-[prometheus]
-  # Whether to export metrics
-  disable = false
-  # HTTP listen address for /metrics endpoint
-  http_listen = "0.0.0.0:{{ unifi_poller_prometheus_port }}"
-  # Path for metrics endpoint
-  report_errors = false
-
-[unifi]
-  # UniFi Controller configurations
-  [[unifi.controller]]
-    # URL for the UniFi Controller
-    url = "{{ unifi_controller_url }}"
-    # Username
-    user = "{{ unifi_controller_user }}"
-    # Password
-    pass = "{{ unifi_controller_pass }}"
-    # Sites to poll (leave empty for all sites)
-    sites = ["{{ unifi_controller_site }}"]
-    # Verify SSL certificates
-    verify_ssl = {{ unifi_controller_verify_ssl | lower }}
-    # Save site information
-    save_sites = {{ unifi_poller_save_sites | lower }}
-    # Save device IDs
-    save_ids = {{ unifi_poller_save_ids | lower }}
-    # Save events
-    save_events = {{ unifi_poller_save_events | lower }}
-    # Save alarms
-    save_alarms = {{ unifi_poller_save_alarms | lower }}
-    # Save anomalies
-    save_anomalies = {{ unifi_poller_save_anomalies | lower }}
-    # Save DPI data
-    save_dpi = {{ unifi_poller_save_dpi | lower }}

ci/images/heating-can/Containerfile

@@ -103,6 +103,8 @@ RUN open3e --help > /dev/null \
  && supervisord --help > /dev/null \
  && grep -q '^\[program:open3e\]' /etc/supervisord.conf \
  && grep -q '^\[program:e3oncan\]' /etc/supervisord.conf \
+ && grep -q '^\[supervisorctl\]' /etc/supervisord.conf \
+ && supervisorctl --help > /dev/null \
  && echo "self-test OK: both services execute; supervisord has both programs"
 
 # CAN_RAW needs no capabilities, so nothing here runs as root.

ci/images/heating-can/supervisord.conf

@@ -10,6 +10,24 @@
 # expands %(ENV_NAME)s at start, so neither the compose file nor a wrapper
 # script constructs open3e's or E3onCAN's argv -- those CLIs stay an
 # implementation detail of this image.
+# Control socket. Without these two sections supervisorctl cannot connect, so
+# there is no way to query or restart an individual service inside the
+# container -- which is the entire reason for running a supervisor rather than
+# one process per container. The first revision shipped without them.
+#
+# Unix socket only, inside the container: nothing outside needs to reach it,
+# and the container runs with host networking, so a TCP listener would be
+# exposed on the host.
+[unix_http_server]
+file=/tmp/supervisor.sock
+chmod=0700
+
+[rpcinterface:supervisor]
+supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
+
+[supervisorctl]
+serverurl=unix:///tmp/supervisor.sock
+
 [supervisord]
 nodaemon=true
 user=open3e

kubernetes/apps/authentik/blueprints-configmap.yaml

@@ -1326,7 +1326,121 @@ data:
           name: Caddy docs.renner.uno
         state: absent
 
-  # Passkey-only (WebAuthn) login for selected apps, in two parts:
+      # ---------------------------------------------------------------
+      # wordpress-gate — www.renner.uno
+      #
+      # Unlike every other gate here, this one does NOT front a whole
+      # host. www.renner.uno is a public family blog; only /wp-admin and
+      # /wp-login.php are routed to this outpost (see
+      # kubernetes/apps/wordpress/ingress.yaml). The front end never
+      # touches it.
+      #
+      # The reverse carve-out — gate everything, skip_path_regex the
+      # public parts, as paperless-gate does for /static/ — is not
+      # possible here: the public set is every post URL, so the pattern
+      # would need a negative lookahead and the outpost's RE2 has none.
+      - id: wordpress-authors-group
+        model: authentik_core.group
+        identifiers:
+          name: WordPress Authors
+        state: present
+        attrs:
+          is_superuser: false
+          # Git owns membership, as with the other gates: this group is the
+          # explicit allowlist for who may publish on www.renner.uno.
+          # Add/remove members HERE, not in the UI — blueprint apply is
+          # authoritative for this M2M and reverts UI edits.
+          #
+          # s.renner is the Authentik account for the site's WordPress
+          # Editor, which still carries the pre-2019 login `s.birzer` — the
+          # two names do not match, and that is expected.
+          users:
+            - !Find [authentik_core.user, [username, k.renner]]
+            - !Find [authentik_core.user, [username, s.renner]]
+
+      - id: wordpress-gate-provider
+        model: authentik_providers_proxy.proxyprovider
+        identifiers:
+          name: wordpress-gate
+        state: present
+        attrs:
+          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          mode: proxy
+          internal_host: http://wordpress.wordpress.svc.cluster.local
+          internal_host_ssl_validation: false
+          external_host: https://www.renner.uno
+          cookie_domain: renner.uno
+          access_token_validity: hours=24
+          refresh_token_validity: days=30
+          intercept_header_auth: true
+          # No skip_path_regex: wp-admin's assets are only ever fetched by an
+          # already-authenticated session, and wp-admin has no manifest or
+          # service worker — the credentials:'omit' PWA problem that forced
+          # carve-outs on vscode-gate and retired music-assistant-gate does
+          # not arise here.
+
+      - model: authentik_core.application
+        identifiers:
+          slug: wordpress-gate
+        state: present
+        attrs:
+          name: WordPress (Gate)
+          slug: wordpress-gate
+          provider: !KeyOf wordpress-gate-provider
+          meta_launch_url: https://www.renner.uno/wp-admin/
+          meta_description: "Forward-auth gate in front of wp-admin on www.renner.uno"
+
+      # Only WordPress Authors pass the gate — see the group above for the
+      # membership list. Both accounts that exist on the migrated site are
+      # covered, so the gate cannot lock out the editor the site is
+      # currently being kept alive for.
+      - model: authentik_policies.policybinding
+        identifiers:
+          target: !Find [authentik_core.application, [slug, wordpress-gate]]
+          group: !KeyOf wordpress-authors-group
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+          failure_result: false
+
+      - model: authentik_outposts.outpost
+        identifiers:
+          name: wordpress-gate
+        state: present
+        attrs:
+          type: proxy
+          service_connection: !KeyOf k8s-conn
+          providers:
+            - !KeyOf wordpress-gate-provider
+          config:
+            log_level: info
+            authentik_host: "http://authentik-server.authentik.svc.cluster.local"
+            authentik_host_insecure: false
+            authentik_host_browser: "https://auth.renner.uno/"
+            refresh_interval: "minutes=5"
+            container_image: null
+            object_naming_template: "ak-outpost-%(name)s"
+            kubernetes_namespace: authentik
+            kubernetes_replicas: 1
+            kubernetes_service_type: ClusterIP
+            kubernetes_disabled_components: ["ingress"]
+            kubernetes_image_pull_secrets: []
+            kubernetes_ingress_annotations: {}
+            kubernetes_ingress_class_name: null
+            kubernetes_ingress_secret_name: "authentik-outpost-tls"
+            kubernetes_json_patches: null
+
+  # Passkey login for selected apps. Passkey is mandatory for accounts
+  # that hold one and unavailable to accounts that do not, so both flows
+  # branch on enrollment (policy passkey-user-has-webauthn): enrolled →
+  # WebAuthn, not enrolled → password + MFA. Enrollment status is a
+  # property of the identified account, so the weaker branch is never
+  # something a requester can select.
+  #
+  # Two parts:
   #
   # 1. A passwordless AUTHENTICATION flow (username → passkey, no
   #    password prompt) bound per-provider, so fresh logins to these
@@ -1335,7 +1449,13 @@ data:
   # 2. A step-up AUTHORIZATION flow — unlike the authentication flow,
   #    this runs on EVERY interactive authorize, so an existing
   #    password-only SSO session from another app cannot slip into
-  #    these apps without presenting a passkey.
+  #    these apps without presenting a passkey. Two things narrow that:
+  #    the stage is skipped outright for accounts with no passkey (they
+  #    have no way to satisfy it), and last_auth_threshold reduces it to
+  #    "without presenting a passkey in this browser within the window"
+  #    — one touch then covers every app, including a later
+  #    password-only session for the same user. The cookie is keyed to
+  #    that user's device, so it never carries across users.
   passkey-stepup.yaml: |
     version: 1
     metadata:
@@ -1356,43 +1476,99 @@ data:
           # flow this replaces.
           authentication: require_authenticated
 
-      - id: passkey-stepup-webauthn-validation
+      # ONE stage object, shared by the step-up authorization flow and
+      # the passwordless authentication flow below.
+      #
+      # `last_auth_threshold` is enforced with a single browser cookie
+      # (`authentik_mfa`) whose JWT is both signed with and carries the
+      # stage's PK, so a validation stage only ever honours a cookie it
+      # wrote itself. Two separate stage objects therefore each demand
+      # their own touch, and — sharing the one cookie name — overwrite
+      # each other, so neither window survives. Sharing this object is
+      # what lets the login touch satisfy the per-app step-up.
+      #
+      # Declared here, ahead of every !KeyOf reference to it: the
+      # blueprint importer resolves entries top-to-bottom.
+      - id: passkey-webauthn-validation
         model: authentik_stages_authenticator_validate.authenticatorvalidatestage
         identifiers:
-          name: passkey-stepup-webauthn-validation
+          name: passkey-webauthn-validation
         state: present
         attrs:
           device_classes:
             - webauthn
-          # `configure`, not `deny`: g.renner has Home Assistant access
-          # but no enrolled WebAuthn device (live check 2026-07-07), so
-          # a hard deny would lock her out with no self-service path.
-          # `configure` injects the default WebAuthn enrollment stage
-          # inline — first login without a passkey becomes "enroll one
-          # now, then validate it". Still passkey-only: there is no way
-          # through this stage without a WebAuthn credential.
-          not_configured_action: configure
-          configuration_stages:
-            - !Find [authentik_stages_authenticator_webauthn.authenticatorwebauthnstage, [name, default-authenticator-webauthn-setup]]
-          # Skip re-validation if a passkey was already presented within
-          # the window. Non-zero for two reasons: (1) no double-touch
-          # when the login itself used a passkey seconds earlier;
-          # (2) NetBird's dashboard silently re-authorizes via
-          # /silent-auth (prompt=none) on token expiry — a flow that
-          # always demands interaction would fail those and force a
-          # visible re-login every access-token lifetime.
+          # `deny`, not `configure`: the authentication flow reaches this
+          # stage with the user merely *identified* (a typed username)
+          # and no password stage in front of it, so inline enrollment
+          # would let anyone who knows a passkey-less username enroll
+          # their own passkey and take over that account. The policy
+          # binding means only accounts that already hold a passkey reach
+          # this stage at all, so `deny` is defence in depth rather than
+          # a path anyone walks — accounts without one branch to password
+          # + MFA instead.
+          not_configured_action: deny
+          # Window within which an already-presented passkey satisfies a
+          # later run of this stage. Non-zero for two reasons: (1) the
+          # per-app step-up must not demand a second touch seconds after
+          # the login touch; (2) NetBird's dashboard silently
+          # re-authorizes via /silent-auth (prompt=none) on token expiry —
+          # a stage that always demands interaction fails those and
+          # forces a visible re-login every access-token lifetime.
+          #
+          # The cookie is browser-scoped, so inside the window a re-login
+          # in that same browser is satisfied by the cookie alone and the
+          # username is all that gets typed. Accepted for a single touch
+          # per app.
           last_auth_threshold: hours=8
           webauthn_user_verification: preferred
 
-      - model: authentik_flows.flowstagebinding
+      # Gate on "this account has a passkey", so enrollment status picks
+      # the branch. Enrolled accounts must use WebAuthn — at login and at
+      # every authorize — while accounts with no passkey fall back to
+      # password + MFA. An attacker cannot pick the weaker branch: the
+      # user is already identified when this runs, so the branch follows
+      # the victim's enrollment, not the requester's choice.
+      #
+      # request.user is the flow's pending_user, not the (anonymous)
+      # requester: ReevaluateMarker builds the engine with
+      # plan.context[PLAN_CONTEXT_PENDING_USER] (flows/markers.py). That
+      # requires the bindings below to keep evaluate_on_plan false and
+      # re_evaluate_policies true — on-plan evaluation happens before the
+      # identification stage has resolved a user.
+      - id: policy-has-passkey
+        model: authentik_policies_expression.expressionpolicy
+        identifiers:
+          name: passkey-user-has-webauthn
+        state: present
+        attrs:
+          expression: |
+            return ak_user_has_authenticator(request.user, "webauthn")
+
+      - id: binding-stepup-webauthn
+        model: authentik_flows.flowstagebinding
         identifiers:
           target: !KeyOf passkey-stepup-flow
-          stage: !KeyOf passkey-stepup-webauthn-validation
+          stage: !KeyOf passkey-webauthn-validation
           order: 10
         state: present
+        attrs:
+          evaluate_on_plan: false
+          re_evaluate_policies: true
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          policy: !KeyOf policy-has-passkey
+          target: !KeyOf binding-stepup-webauthn
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+          failure_result: false
 
       # Passwordless authentication flow: identification (username or
-      # email, NO password stage) → WebAuthn validation → login.
+      # email, NO password stage) → WebAuthn validation → login, with a
+      # password + MFA branch for accounts that have no passkey.
       - id: passkey-passwordless-flow
         model: authentik_flows.flow
         identifiers:
@@ -1403,37 +1579,6 @@ data:
           title: "Sign in with your passkey"
           designation: authentication
 
-      # SEPARATE stage from the step-up one, and deliberately `deny`:
-      # at this point in an authentication flow the user has only been
-      # *identified* (typed a username), not authenticated — `configure`
-      # here would let anyone who knows a passkey-less username enroll
-      # their own passkey and take over that account. Inline enrollment
-      # for passkey-less users happens exclusively in the authorization
-      # step-up stage above, where require_authenticated guarantees a
-      # fully authenticated session. A passkey-less user therefore
-      # bootstraps via password login on auth.renner.uno, then gets the
-      # enrollment prompt on first app access.
-      #
-      # Declared BEFORE the identification stage because the latter
-      # references it via !KeyOf (webauthn_stage), and the blueprint
-      # importer resolves entries top-to-bottom.
-      - id: passkey-passwordless-webauthn-validation
-        model: authentik_stages_authenticator_validate.authenticatorvalidatestage
-        identifiers:
-          name: passkey-passwordless-webauthn-validation
-        state: present
-        attrs:
-          device_classes:
-            - webauthn
-          not_configured_action: deny
-          # Small non-zero window so a passkey presented via the
-          # identification stage's autofill (webauthn_stage below) is
-          # not challenged a second time by this stage at order 20 in
-          # the same flow run. The skip is session-scoped, so a fresh
-          # (attacker) session never inherits it.
-          last_auth_threshold: minutes=5
-          webauthn_user_verification: preferred
-
       # webauthn_stage enables passkey autofill (conditional WebAuthn,
       # authentik >= 2025.12): the browser offers enrolled passkeys in
       # the username field's autofill — one tap, no username typed.
@@ -1449,7 +1594,7 @@ data:
           user_fields:
             - username
             - email
-          webauthn_stage: !KeyOf passkey-passwordless-webauthn-validation
+          webauthn_stage: !KeyOf passkey-webauthn-validation
 
       - model: authentik_flows.flowstagebinding
         identifiers:
@@ -1458,12 +1603,87 @@ data:
           order: 10
         state: present
 
-      - model: authentik_flows.flowstagebinding
+      # Order 20 and 21/22 are the two mutually exclusive branches.
+      - id: binding-passwordless-webauthn
+        model: authentik_flows.flowstagebinding
         identifiers:
           target: !KeyOf passkey-passwordless-flow
-          stage: !KeyOf passkey-passwordless-webauthn-validation
+          stage: !KeyOf passkey-webauthn-validation
           order: 20
         state: present
+        attrs:
+          evaluate_on_plan: false
+          re_evaluate_policies: true
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          policy: !KeyOf policy-has-passkey
+          target: !KeyOf binding-passwordless-webauthn
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+          failure_result: false
+
+      # The negated branch. failure_result: true is the safety net — the
+      # UserLoginStage defaults a missing authentication backend to
+      # BACKEND_INBUILT instead of refusing (user_login/stage.py), so a
+      # flow that drops BOTH branches would log the user in on a typed
+      # username alone. A PolicyException is resolved to failure_result
+      # before negate is applied (policies/process.py), so a broken
+      # expression pulls the password stage IN and degrades everyone to
+      # password + MFA rather than to no factor at all.
+      - id: binding-passwordless-password
+        model: authentik_flows.flowstagebinding
+        identifiers:
+          target: !KeyOf passkey-passwordless-flow
+          stage: !Find [authentik_stages_password.passwordstage, [name, default-authentication-password]]
+          order: 21
+        state: present
+        attrs:
+          evaluate_on_plan: false
+          re_evaluate_policies: true
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          policy: !KeyOf policy-has-passkey
+          target: !KeyOf binding-passwordless-password
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: true
+          failure_result: true
+
+      # Same MFA validation the default authentication flow runs at its
+      # order 30, so the fallback is not a weaker way into these apps
+      # than auth.renner.uno's own login already is. It carries
+      # not_configured_action: configure, which is safe here and nowhere
+      # else in this flow: the password stage above has established a
+      # first factor by the time it runs, so it doubles as the enrollment
+      # path for accounts holding no authenticator at all.
+      - id: binding-passwordless-mfa
+        model: authentik_flows.flowstagebinding
+        identifiers:
+          target: !KeyOf passkey-passwordless-flow
+          stage: !Find [authentik_stages_authenticator_validate.authenticatorvalidatestage, [name, default-authentication-mfa-validation]]
+          order: 22
+        state: present
+        attrs:
+          evaluate_on_plan: false
+          re_evaluate_policies: true
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          policy: !KeyOf policy-has-passkey
+          target: !KeyOf binding-passwordless-mfa
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: true
+          failure_result: true
 
       - model: authentik_flows.flowstagebinding
         identifiers:
@@ -1577,3 +1797,19 @@ data:
           redirect_uris:
             - url: https://dash.renner.uno/login/generic_oauth
               matching_mode: strict
+
+      # Superseded by the single passkey-webauthn-validation stage above.
+      # Their flowstagebindings are FK-CASCADEd away with them; the
+      # identification stage's webauthn_stage FK is SET_NULL and is
+      # re-pointed above, so no dangling reference survives. Declared
+      # last: the importer applies entries top-to-bottom inside one
+      # transaction, so the replacements exist before these go.
+      - model: authentik_stages_authenticator_validate.authenticatorvalidatestage
+        identifiers:
+          name: passkey-stepup-webauthn-validation
+        state: absent
+
+      - model: authentik_stages_authenticator_validate.authenticatorvalidatestage
+        identifiers:
+          name: passkey-passwordless-webauthn-validation
+        state: absent

kubernetes/apps/authentik/kustomization.yaml

@@ -14,6 +14,7 @@ resources:
   - zone-configurator-gate-egress.yaml
   - alertmanager-gate-egress.yaml
   - prometheus-gate-egress.yaml
+  - wordpress-gate-egress.yaml
   - outpost-servicemonitors.yaml
   - ldap-outpost-cert.yaml
   - ldap-outpost-lb.yaml

kubernetes/apps/authentik/wordpress-gate-egress.yaml

@@ -0,0 +1,28 @@
+---
+# Allow the dedicated wordpress-gate outpost to egress into the wordpress
+# namespace and reach WordPress (:80). Without this, the authentik-namespace
+# baseline CNP has no egress rule to wordpress, so the gate outpost's
+# forwarded requests are dropped. Scoped to the gate outpost's pod label so no
+# other authentik component can reach it. Mirrors esphome-gate-egress.yaml.
+#
+# Note this gate only ever sees /wp-admin and /wp-login.php — the public site
+# is routed straight to the same Service by a separate Ingress, so this rule
+# is not the only path into the wordpress namespace, unlike the paperless case.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: wordpress-gate-egress
+  namespace: authentik
+spec:
+  endpointSelector:
+    matchLabels:
+      goauthentik.io/outpost-name: wordpress-gate
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": wordpress
+            app.kubernetes.io/name: wordpress
+      toPorts:
+        - ports:
+            - port: "80"
+              protocol: TCP

kubernetes/apps/gatus/configmap.yaml

@@ -160,6 +160,24 @@ data:
         conditions:
           - "[STATUS] == any(200, 301, 302)"
 
+      # Terminus admin UI. Checks the TLS route only — the cleartext
+      # device endpoint (trmnl-device.renner.uno) is the same pod, so a
+      # failure here covers both. Terminus exposes no health endpoint, so
+      # this is a root-page check: the login page answers 200, and an
+      # unauthenticated request may redirect.
+      - name: Web Service 10
+        group: services
+        url: https://trmnl.renner.uno
+        interval: 60s
+        ui:
+          hide-hostname: true
+          hide-url: true
+          hide-port: true
+          hide-conditions: true
+          hide-errors: true
+        conditions:
+          - "[STATUS] == any(200, 301, 302)"
+
       # Removed 2026-05-21: Jellyfin (media.renner.uno), Wiki (wiki.renner.uno),
       # InfluxDB (influx.renner.uno) — hostnames NXDOMAIN, no Ingress; services
       # no longer deployed. Re-add when/if they come back.

kubernetes/apps/home-assistant/deployment.yaml

@@ -38,7 +38,7 @@ spec:
       # gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
       containers:
         - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.8.1 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
           env:
             - name: TZ
               value: Europe/Berlin

kubernetes/apps/ical-proxy/deployment.yaml

@@ -17,7 +17,33 @@ spec:
     spec:
       containers:
         - name: ical-proxy
-          image: ghcr.io/konairius/ical-proxy:latest
+          # Pinned to a commit build rather than :latest.
+          #
+          # :latest never rolls. The tag moves when a build publishes, but
+          # nothing tells the Deployment about it: the pod template is
+          # unchanged, so Kubernetes has no reason to replace anything, and
+          # this image is deliberately NOT enrolled in Flux image automation
+          # — see kubernetes/infrastructure/image-automation/README.md, which
+          # lists ghcr.io/konairius/ical-proxy as having "only main, latest,
+          # and main-<sha> commit builds; no release tags", so there is no
+          # ImagePolicy watching it. The pods therefore sat on a 21-day-old
+          # image while :latest moved underneath them, and the only way to
+          # pick up a new build was a manual restart.
+          #
+          # A pinned tag makes the rollout a git change: Flux applies the new
+          # pod template and Kubernetes rolls it, which is also what makes
+          # what is RUNNING recoverable from this repo rather than "whatever
+          # :latest meant the last time a pod happened to restart".
+          #
+          # main-99a912b is the merge of konairius/ical-proxy#12, which adds
+          # the summary_match/summary_replace parameters this deployment
+          # needs to strip "| Abfuhrkalender - Landkreis Amberg-Sulzbach"
+          # from every waste collection.
+          #
+          # Bumping this is a manual, reviewed edit. The tag format is
+          # main-<short-sha> (docker/metadata-action type=sha with the branch
+          # as prefix).
+          image: ghcr.io/konairius/ical-proxy:main-99a912b
           ports:
             - containerPort: 8080
               name: http

kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml

@@ -73,3 +73,29 @@ spec:
       interval: 30s
   podTargetLabels:
     - role
+---
+# wp-pg, restored into the `woodpecker` namespace (issue #433). It has never
+# been scraped: `apps` had no PodMonitor for it AND its baseline network policy
+# never allowed the monitoring namespace, so `cnpg_collector_up` has series for
+# the four clusters above but not this one. The namespace move fixes the policy
+# half; without this PodMonitor there is still no scrape target at all.
+#
+# Deliberately not adding one for the cluster still in `apps`: it is retired in
+# stage 3, and a PodMonitor there would only be reconciled to be deleted again.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: cnpg-wp-pg
+  namespace: woodpecker
+  labels:
+    app.kubernetes.io/name: cloudnative-pg
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: wp-pg
+  podMetricsEndpoints:
+    - port: metrics
+      interval: 30s
+  podTargetLabels:
+    - role

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -4,6 +4,8 @@ kind: Kustomization
 # come from the kube-prometheus-stack chart. Flux applies this Kustomization
 # only after `monitoring` is Ready (chart installed → CRDs present).
 resources:
+  - prometheusrule-ceph.yaml
+  - prometheusrule-cilium.yaml
   - prometheusrule-hetzner-backup.yaml
   - prometheusrule-loki.yaml
   - prometheusrule-netbird-availability.yaml

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml

@@ -0,0 +1,56 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: ceph
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: ceph.rules
+      rules:
+        - alert: CephHealthWarn
+          # ceph_health_status: 0 = HEALTH_OK, 1 = HEALTH_WARN, 2 = HEALTH_ERR.
+          # 6h is deliberately long. Ceph warns for self-clearing conditions —
+          # a slow BlueStore op during the Sunday restore sweep, a mon briefly
+          # below its disk threshold — and paging on those trains people to
+          # ignore the alert. What matters is a warning that does not clear.
+          expr: ceph_health_status == 1
+          for: 6h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Ceph has been HEALTH_WARN for 6h"
+            description: "The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
+
+        - alert: CephHealthError
+          # HEALTH_ERR means data availability or redundancy is at risk.
+          expr: ceph_health_status == 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Ceph is HEALTH_ERR"
+            description: "The Ceph cluster is in HEALTH_ERR — availability or redundancy is at risk. Inspect immediately with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` and `ceph osd df tree`."
+
+        - alert: CephMonDiskLow
+          # Named explicitly rather than left to CephHealthWarn, because the
+          # mons sit on the node's /var alongside container storage: this
+          # fires for a full node, not for Ceph growth. The mon's own data is
+          # ~106 MB, so 30% of a 60 GB shared filesystem says nothing about
+          # Ceph.
+          # Image GC does not clear this — measured on k8s-2, the reclaimable
+          # content store is a small fraction of /var/lib/containerd next to
+          # ~20 GB of overlayfs snapshots held by running containers. Look at
+          # what actually grew before reaching for a threshold.
+          expr: |
+            100 * node_filesystem_avail_bytes{mountpoint="/var"}
+              / node_filesystem_size_bytes{mountpoint="/var"} < 30
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ $labels.instance }} /var below 30% free — Ceph mon will warn"
+            description: "/var on {{ $labels.instance }} is under 30% free, which is Ceph's mon_data_avail_warn threshold, so the mon on this node will report MON_DISK_LOW. Find the actual consumer with `talosctl -n <node> usage /var/lib --depth 1` and then `/var/lib/containerd --depth 1` — note that overlayfs snapshots belong to running containers and are not reclaimable by image GC."

kubernetes/apps/monitoring-rules/prometheusrule-cilium.yaml

@@ -0,0 +1,44 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: cilium
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: cilium.rules
+      # Both alerts are map-agnostic on purpose. cilium_bpf_map_pressure is
+      # emitted per map type — policy, conntrack, NAT, LB — distinguished by
+      # the map_name label, and running out of any of them degrades the
+      # datapath. The policy map is merely the one that has bitten us.
+      # Cilium only exports the metric for maps above
+      # policyMapPressureMetricsThreshold (chart default 0.1), so quiet maps
+      # produce no series rather than a flood of near-zero ones.
+      rules:
+        - alert: CiliumBPFMapPressureHigh
+          # cilium_bpf_map_pressure is used/capacity for the fullest map of
+          # that name on the agent. A full map is not a slow path — it is a
+          # hard failure: the agent cannot program the datapath, so endpoint
+          # regeneration fails and every new pod on the node hangs in sandbox
+          # creation until something frees space.
+          expr: cilium_bpf_map_pressure > 0.8
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Cilium BPF map {{ $labels.map_name }} is {{ $value | humanizePercentage }} full on {{ $labels.node }}"
+            description: "The BPF map {{ $labels.map_name }} on node {{ $labels.node }} has been over 80% full for 15 minutes. At 100% the Cilium agent can no longer program the datapath and new pods on this node fail sandbox creation. Raise the corresponding size in kubernetes/infrastructure/cilium/helmrelease.yaml (values.bpf) — the map is sized at creation, so the change needs an agent restart."
+
+        - alert: CiliumBPFMapFull
+          # Pressure reaches (and can exceed) 1.0 when the agent wants more
+          # entries than the map holds. Pods are already failing to start.
+          expr: cilium_bpf_map_pressure >= 1
+          for: 0m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Cilium BPF map {{ $labels.map_name }} is full on {{ $labels.node }}"
+            description: "The BPF map {{ $labels.map_name }} on node {{ $labels.node }} is at or over capacity. Endpoint regeneration is failing with 'no space left on device' and pods scheduled to this node will hang in ContainerCreating. Check `{namespace=\"kube-system\",pod=~\"cilium-.+\"} |= \"no space left on device\"` in Loki for the affected endpoints."

kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml

@@ -12,37 +12,37 @@ spec:
   groups:
     - name: woodpecker-rotation.rules
       # Convergence safety net for the automated WOODPECKER_AGENT_SECRET rotation
-      # (kubernetes/apps/woodpecker/rotate-agent-secret.yaml). That rotation Job is
+      # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml). That rotation Job is
       # fire-and-forget: it writes OpenBao and exits WITHOUT verifying ESO re-synced
       # or that both consumers restarted healthy. These alerts close that gap. ESO
       # metrics are not scraped in this cluster, so everything here is built on
-      # kube-state-metrics. The CronJob/Job series stay empty until the rotation
-      # feature (branch feat/woodpecker-secret-auto-rotation) is merged — harmless.
+      # kube-state-metrics. The rotation has since landed: the CronJob lives in the
+      # openbao namespace, not apps, which is where the last two rules look.
       rules:
         - alert: WoodpeckerServerNotReady
           # 0 ready replicas well past the 10m ESO refresh + Reloader restart cycle.
-          expr: kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"} == 0
+          expr: kube_deployment_status_replicas_ready{namespace="woodpecker",deployment="woodpecker-server"} == 0
           for: 20m
           labels:
             severity: critical
           annotations:
             summary: "Woodpecker CI server has no ready replica"
             description: >-
-              StatefulSet woodpecker-server (ns apps) has had 0 ready replicas for 20m;
-              all CI is down (agents cannot reach woodpecker.apps.svc:9000). If this
-              followed a secret rotation, the server pod failed to return after its
-              Reloader-triggered restart — check `kubectl -n apps describe pod
-              woodpecker-server-0` (PVC attach / crashloop) and the woodpecker-secret
-              ExternalSecret sync status.
+              Deployment woodpecker-server (ns woodpecker) has had 0 ready replicas
+              for 20m; all CI is down (agents cannot reach
+              woodpecker.woodpecker.svc:9000). If this followed a secret rotation, the
+              server pod failed to return after its Reloader-triggered restart — check
+              `kubectl -n woodpecker describe deploy woodpecker-server` and the
+              woodpecker-secret ExternalSecret sync status.
         - alert: WoodpeckerAgentNotReady
-          expr: kube_deployment_status_replicas_available{namespace="apps",deployment="woodpecker-agent"} == 0
+          expr: kube_deployment_status_replicas_available{namespace="woodpecker",deployment="woodpecker-agent"} == 0
           for: 20m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker CI agent has no available replica"
             description: >-
-              Deployment woodpecker-agent (ns apps) has had 0 available replicas for
+              Deployment woodpecker-agent (ns woodpecker) has had 0 available replicas for
               20m; no pipelines will run. If this followed a secret rotation, the agent
               did not return after its Reloader restart, or its WOODPECKER_AGENT_SECRET
               no longer matches the server's — check the agent pod logs for a gRPC auth
@@ -50,29 +50,29 @@ spec:
         - alert: WoodpeckerSecretRotationJobFailed
           # Named, clearer signal than the generic KubeJobFailed for the OpenBao-write
           # step. Fires while a failed rotation Job is retained (ttl 3d).
-          expr: kube_job_status_failed{namespace="apps",job_name=~"woodpecker-secret-rotate.*"} > 0
+          expr: kube_job_status_failed{namespace="openbao",job_name=~"woodpecker-secret-rotate.*"} > 0
           for: 15m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent-secret rotation Job failed"
             description: >-
-              A woodpecker-secret-rotate Job (ns apps) failed at the OpenBao-write step;
+              A woodpecker-secret-rotate Job (ns openbao) failed at the OpenBao-write step;
               the agent secret was NOT rotated. Most common cause: the one-time OpenBao
               `rotator` role/policy bootstrap was never applied (see
               kubernetes/infrastructure/openbao/policies/rotator.hcl). Check
-              `kubectl -n apps logs job/<name>`.
+              `kubectl -n openbao logs job/<name>`.
         - alert: WoodpeckerSecretRotationStale
           # Quarterly cadence (~91d) + grace. The series is ABSENT until the first
           # successful run, so this cannot false-fire before rotation has ever worked.
-          expr: (time() - kube_cronjob_status_last_successful_time{namespace="apps",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
+          expr: (time() - kube_cronjob_status_last_successful_time{namespace="openbao",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
           for: 1h
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent secret has not rotated in >100 days"
             description: >-
-              The quarterly woodpecker-secret-rotate CronJob (ns apps) has not completed
+              The quarterly woodpecker-secret-rotate CronJob (ns openbao) has not completed
               successfully in over 100 days — rotation has silently stopped (CronJob
               suspended/deleted, or every run failing). Investigate the CronJob and its
               recent Jobs.

kubernetes/apps/music-assistant/deployment.yaml

@@ -52,7 +52,7 @@ spec:
         # stdlib only; Flux keeps the tag in lockstep via the shared image
         # policy) — no extra dependency.
         - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.11 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
           command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
           volumeMounts:
             - name: data
@@ -62,7 +62,7 @@ spec:
               readOnly: true
       containers:
         - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.11 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
           env:
             - name: LOG_LEVEL
               value: info

kubernetes/apps/paperless/gotenberg.yaml

@@ -22,7 +22,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: gotenberg
-          image: gotenberg/gotenberg:8.35.0 # {"$imagepolicy": "flux-system:gotenberg"}
+          image: gotenberg/gotenberg:8.36.0 # {"$imagepolicy": "flux-system:gotenberg"}
           args:
             - "gotenberg"
             - "--chromium-disable-javascript=true"

kubernetes/apps/trmnl/cluster.yaml

@@ -0,0 +1,41 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: tr-pg
+  namespace: trmnl
+spec:
+  instances: 1
+  smartShutdownTimeout: 10
+  stopDelay: 60
+  # PG 18, NOT the repo-standard 16. Terminus ships a committed
+  # config/db/structure.sql containing `SET transaction_timeout = 0;` — a GUC
+  # that does not exist before PostgreSQL 17, so schema load against a 16
+  # server dies with `unrecognized configuration parameter`. Upstream's
+  # compose.yml develops against 18.4, so 18 is the tested target; 17 is the
+  # bare minimum. Bare major tag matches the other clusters (minor security
+  # bumps flow automatically, major upgrades stay an explicit decision).
+  imageName: ghcr.io/cloudnative-pg/postgresql:18
+  enableSuperuserAccess: true
+  storage:
+    storageClass: ceph-block
+    size: 5Gi
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      isWALArchiver: true
+      parameters:
+        barmanObjectName: tr-pg-store
+  bootstrap:
+    initdb:
+      database: terminus
+      owner: terminus
+  postgresql:
+    parameters:
+      shared_buffers: "128MB"
+      max_connections: "50"
+      wal_compression: "on"
+  resources:
+    requests:
+      cpu: 100m
+      memory: 256Mi
+    limits:
+      memory: 512Mi

kubernetes/apps/trmnl/device-service.yaml

@@ -0,0 +1,48 @@
+# Device-facing endpoint. Cleartext HTTP, deliberately.
+#
+# The TRMNL firmware initiates every request and cannot be handed the
+# internal CA, so terminating TLS here would simply break polling. Scope of
+# the exposure: this Service is LAN-only (no public DNS label, no Cloudflare
+# record), it carries no admin session — the device authenticates with its
+# MAC in the `ID` header plus a per-device api_key — and the Terminus admin
+# UI stays on the TLS route at trmnl.renner.uno.
+#
+# Same shape as home-assistant-bgp: a dedicated address on the Cilium BGP
+# pool rather than a TLS-less Ingress, which would be a new pattern for this
+# repo. The device is provisioned with the HOSTNAME, not the address.
+#
+# The address is requested explicitly, matching every other LB service here.
+# Left to allocate freely, Cilium IPAM handed out the first free address in
+# the pool block (10.2.99.0-10.2.99.255) — which is 10.2.99.0, the base
+# address of the range. Each LB address is advertised to the UDM as its own
+# BGP route so that is routable in principle, but a .0 host address is the
+# kind of thing constrained embedded IP stacks get wrong, and the TRMNL
+# firmware is the single client that has to work here. Not worth the risk
+# for an address nobody chose on purpose.
+#
+# .89 continues the existing sequence (.81-.88 are in use) and matches the
+# fd99::89 that IPAM had already picked for v6.
+apiVersion: v1
+kind: Service
+metadata:
+  name: terminus-device
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus
+    lb-pool: bgp
+  annotations:
+    external-dns.alpha.kubernetes.io/hostname: trmnl-device.renner.uno
+    lbipam.cilium.io/ips: "10.2.99.89,fd99::89"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Local
+  ipFamilyPolicy: PreferDualStack
+  ipFamilies:
+    - IPv4
+    - IPv6
+  selector:
+    app.kubernetes.io/name: terminus
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

kubernetes/apps/trmnl/externalsecret.yaml

@@ -0,0 +1,18 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: terminus-secrets
+  namespace: trmnl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: terminus-secrets
+  # Holds APP_SECRET only (session/cookie/CSRF key). DATABASE_URL is not in
+  # here — it comes straight off the CNPG-generated `tr-pg-app` Secret's
+  # `uri` key, so the password rotates with the cluster and is never copied.
+  dataFrom:
+    - extract:
+        key: apps/trmnl

kubernetes/apps/trmnl/hetzner-creds-externalsecret.yaml

@@ -0,0 +1,15 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: trmnl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    - extract:
+        key: seed/trmnl/hetzner-creds

kubernetes/apps/trmnl/ingress.yaml

@@ -0,0 +1,28 @@
+# Admin UI only — login, device registration, playlists, screen previews.
+# The device never touches this route: the firmware speaks plain HTTP and
+# would reject the internal CA, so it uses the cleartext LoadBalancer in
+# device-service.yaml instead. Both front the same Service.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: terminus
+  namespace: trmnl
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - trmnl.renner.uno
+      secretName: trmnl-tls
+  rules:
+    - host: trmnl.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: terminus
+                port:
+                  number: 80

kubernetes/apps/trmnl/kustomization.yaml

@@ -0,0 +1,20 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - externalsecret.yaml
+  - pvcs.yaml
+  - valkey.yaml
+  - terminus.yaml
+  - ingress.yaml
+  - device-service.yaml
+  # Headless HA screenshotter. Lives in this namespace because Terminus
+  # FETCHES rendered screens back from it (uri delivery mode), so the
+  # two need the intra-namespace path.
+  - renderer.yaml
+  - renderer-externalsecret.yaml
+  # tr-pg CNPG Postgres + offsite Barman backup. Note this cluster runs
+  # PostgreSQL 18, not the repo-standard 16 — see the comment in cluster.yaml.
+  - cluster.yaml
+  - objectstore.yaml
+  - scheduled-backup.yaml
+  - hetzner-creds-externalsecret.yaml

kubernetes/apps/trmnl/objectstore.yaml

@@ -0,0 +1,30 @@
+---
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: tr-pg-store
+  namespace: trmnl
+spec:
+  configuration:
+    # Offsite Hetzner Object Storage (fsn1), same bucket as the other CNPG
+    # clusters. No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
+    # (plugin #646), so at-rest encryption is unavailable; transit is TLS.
+    destinationPath: s3://renner-uno-k8s-cnpg/tr-pg
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      # Region is required (REGION=fsn1) — without it boto3's HeadBucket
+      # (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
+  retentionPolicy: "30d"

kubernetes/apps/trmnl/pvcs.yaml

@@ -0,0 +1,57 @@
+---
+# Uploads + synchronised fonts for the Terminus pod. Mounted twice by
+# subPath (uploads/, fonts/) into both the web and sidekiq containers, which
+# share one pod — so RWO is sufficient and the Deployment uses `Recreate`.
+#
+# Contents: rendered device screens and the font set pulled by the
+# FONT_SYNCHRONIZER job. Persisting fonts here (rather than an emptyDir)
+# means a pod restart does not depend on upstream being reachable.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: terminus-data
+  namespace: trmnl
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 5Gi
+---
+# Valkey RDB snapshots. The Sidekiq queue is reconstructable — the
+# synchroniser jobs re-enqueue on boot — so this is small and losing it is
+# an inconvenience, not data loss.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: terminus-valkey-data
+  namespace: trmnl
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 1Gi
+---
+# Renderer state: schedules.json plus the rendered screens under
+# output/, which Terminus fetches back over HTTP in uri delivery mode.
+#
+# This mount is REQUIRED, not just nice to have. The app picks its data
+# directory by testing whether /data exists — with nothing mounted it
+# silently falls back to /app/data inside the container, so schedules
+# and rendered output would vanish on every restart and the panel would
+# go blank until the next render.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: trmnl-renderer-data
+  namespace: trmnl
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 2Gi

kubernetes/apps/trmnl/renderer-externalsecret.yaml

@@ -0,0 +1,27 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: trmnl-renderer-secrets
+  namespace: trmnl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: trmnl-renderer-secrets
+  # Kept separate from apps/trmnl (which holds Terminus's APP_SECRET)
+  # because these are credentials the RENDERER holds about other
+  # systems, with different rotation stories:
+  #
+  #   HA_ACCESS_TOKEN     — a dedicated Home Assistant long-lived token,
+  #                         not the operator's personal one, so the
+  #                         renderer can be revoked in isolation.
+  #   BYOS_ACCESS_TOKEN   — Terminus JWTs, minted once out of band from
+  #   BYOS_REFRESH_TOKEN    a Terminus login. The PASSWORD deliberately
+  #                         never enters the cluster; stock Terminus
+  #                         disables session expiry, so these are
+  #                         effectively permanent.
+  dataFrom:
+    - extract:
+        key: apps/trmnl/renderer

kubernetes/apps/trmnl/renderer.yaml

@@ -0,0 +1,385 @@
+# Headless renderer: screenshots the Home Assistant e-ink dashboard,
+# dithers it to 16 greys and hands it to Terminus.
+#
+# Upstream ships this as a Home Assistant add-on. Our HA is a plain
+# container with no Supervisor, so it runs standalone here instead —
+# viable because the app prefers HOME_ASSISTANT_URL/ACCESS_TOKEN env
+# vars over /data/options.json and authenticates with an ordinary
+# long-lived token. SUPERVISOR_TOKEN is deliberately NOT set: it only
+# flips an internal isAddOn flag, and setting it would switch the app
+# to add-on behaviour (fixed port, /data assumptions).
+#
+# DIRECTION OF TRAVEL, which is the non-obvious part: with the
+# byos-hanami webhook in `uri` delivery mode the renderer does NOT
+# push image bytes. It POSTs a URL pointing at its own /output/<file>
+# endpoint, and TERMINUS FETCHES THE IMAGE BACK from it. That is why
+# both live in this namespace — the pull is intra-namespace and is
+# already permitted by the default-deny-ingress rule.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: trmnl-renderer-schedule
+  namespace: trmnl
+data:
+  # Template for /data/schedules.json. The init container splices the
+  # BYOS tokens in; everything non-secret is declared here so the
+  # schedule is reviewable in git rather than clicked into the app's
+  # web UI (which has no authentication and is not exposed).
+  #
+  # The app re-reads schedules.json every 60 s, so edits here take
+  # effect without a restart once the init container has rewritten it.
+  #
+  # lang is "de", not null. It calls Home Assistant's own
+  # _selectLanguage() for the screenshot session, so the rendered page
+  # is localised by HA rather than by hand. With it left null the
+  # dashboard grew a German weekday array, a German month array and a
+  # fifteen-entry weather-condition dict — reimplementing translations
+  # HA already ships. The instance itself stays language "en"; this
+  # only affects the render session.
+  #
+  # model_id is 1 — TRMNL X (name "v2", 1872x1404, 16 colours) — and it
+  # is an INTEGER, not a name. Confirmed against the running server:
+  # GET /api/screens returns model_id as a number, and upstream's
+  # example value "og_1bit" matches no model on this instance (the
+  # 2-bit OG model here is id 11, name "og_plus"). Getting this wrong
+  # shows up as Terminus rejecting the pushed screen with a 422.
+  #
+  # All three auth fields below are placeholders the init container
+  # replaces — including obtained_at, which must end up a real epoch-MS
+  # stamp. The 0 here is inert only because it is always overwritten;
+  # left as 0 the scheduler reads the tokens as ancient and refuses to
+  # push at all ("expired beyond the refresh window"), with no render
+  # attempt and no error beyond one startup warning.
+  schedules.template.json: |
+    [
+      {
+        "id": "trmnl-entrance",
+        "name": "Entrance e-ink panel",
+        "enabled": true,
+        "cron": "* * * * *",
+        "interval_minutes": 1,
+        "ha_mode": true,
+        "dashboard_path": "/trmnl-entrance/home",
+        "target_url": null,
+        "viewport": { "width": 1872, "height": 1404 },
+        "device": "schedule_trmnl_x",
+        "crop": { "enabled": false, "x": 0, "y": 0, "width": 0, "height": 0 },
+        "format": "png",
+        "rotate": null,
+        "zoom": 1,
+        "wait": null,
+        "theme": "Graphite E-ink Light",
+        "lang": "de",
+        "dark": false,
+        "invert": false,
+        "timestamp": false,
+        "dithering": {
+          "enabled": true,
+          "method": "floyd-steinberg",
+          "palette": "gray-16",
+          "bitDepth": 4,
+          "gammaCorrection": true,
+          "blackLevel": 0,
+          "whiteLevel": 100,
+          "normalize": true,
+          "saturationBoost": false
+        },
+        "webhook_url": "http://terminus.trmnl.svc.cluster.local/api/screens",
+        "webhook_headers": {},
+        "webhook_format": {
+          "format": "byos-hanami",
+          "byosConfig": {
+            "label": "Eingang",
+            "name": "trmnl-entrance",
+            "model_id": 1,
+            "preprocessed": true,
+            "addon_base_url": "http://trmnl-renderer.trmnl.svc.cluster.local:10000",
+            "delivery_mode": "uri",
+            "auth": {
+              "enabled": true,
+              "access_token": "__BYOS_ACCESS_TOKEN__",
+              "refresh_token": "__BYOS_REFRESH_TOKEN__",
+              "obtained_at": 0
+            }
+          }
+        },
+        "createdAt": "2026-08-11T00:00:00.000Z",
+        "updatedAt": "2026-08-11T00:00:00.000Z"
+      }
+    ]
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: trmnl-renderer
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: trmnl-renderer
+  annotations:
+    # BOTH sources need this, and for the same reason: the init
+    # container writes /data/schedules.json ONCE, at pod start. Nothing
+    # re-reads either source afterwards.
+    #
+    # The app re-reads schedules.json every 60s, which makes it look
+    # self-updating — but only the FILE, and the file is only rewritten
+    # by the init container. So a ConfigMap or Secret change lands in
+    # git, updates the object, and the running renderer keeps the old
+    # values indefinitely.
+    #
+    # Both halves were hit repeatedly in practice and each time needed
+    # `kubectl delete pod` by hand: the theme name, the schedule's lang
+    # field, and every re-mint of the BYOS tokens. A fix that depends on
+    # someone remembering to restart a pod is not deployed.
+    #
+    # Named per-resource rather than the blanket
+    # reloader.stakater.com/auto: "true", matching terminus.yaml and
+    # mosquitto-deployment.yaml.
+    configmap.reloader.stakater.com/reload: trmnl-renderer-schedule
+    secret.reloader.stakater.com/reload: trmnl-renderer-secrets
+spec:
+  replicas: 1
+  # RWO PVC, and two renderers would push conflicting screens for the
+  # same device.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: trmnl-renderer
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: trmnl-renderer
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      initContainers:
+        # Writes /data/schedules.json from the ConfigMap template with
+        # the BYOS tokens injected.
+        #
+        # jq rather than sed: the tokens are JWTs and go into a JSON
+        # string, so they need JSON-safe escaping. A sed substitution
+        # would corrupt on any character sed treats as special.
+        #
+        # No HTTP here on purpose. The app does expose a /api/byos/login
+        # proxy that exchanges a Terminus username+password for these
+        # tokens, but that would mean shipping the Terminus PASSWORD
+        # into the cluster and doing a network call at every pod start.
+        # Tokens are therefore minted once out of band and kept in
+        # OpenBao — the password never enters the cluster.
+        #
+        # That only works because Terminus is configured with
+        # API_ACCESS_TOKEN_PERIOD = 10 years (see terminus.yaml). It is
+        # NOT, as an earlier version of this comment claimed, because
+        # stock Terminus issues permanent tokens — the stock default is
+        # 1800s, and this deployment demonstrated it by having the
+        # webhook start failing with "expired JWT access token" about
+        # thirty minutes after seeding.
+        - name: render-schedule
+          image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              # obtained_at is stamped at write time, in MILLISECONDS.
+              # It is not cosmetic: the scheduler derives token age from
+              # it, and a literal 0 made the app treat freshly-minted
+              # tokens as ancient and skip the webhook entirely —
+              #   "BYOS tokens ... expired beyond the refresh window"
+              #
+              # Stamping it "now" is only safe because Terminus is
+              # configured with API_ACCESS_TOKEN_PERIOD = 10 years (see
+              # terminus.yaml). Against the 30-minute default it would be
+              # actively harmful: the app would believe a stale token had
+              # 30 fresh minutes left, skip its proactive refresh, and
+              # then be unable to recover, because refreshing requires a
+              # still-valid token. If that setting is ever reverted, this
+              # must become a real mint time carried alongside the tokens.
+              jq --arg at "$BYOS_ACCESS_TOKEN" \
+                 --arg rt "$BYOS_REFRESH_TOKEN" \
+                 '(.[].webhook_format.byosConfig.auth.access_token) = $at
+                  | (.[].webhook_format.byosConfig.auth.refresh_token) = $rt
+                  | (.[].webhook_format.byosConfig.auth.obtained_at) = (now * 1000 | floor)' \
+                 /template/schedules.template.json > /data/schedules.json
+              echo "wrote /data/schedules.json ($(wc -c < /data/schedules.json) bytes)"
+          env:
+            - name: BYOS_ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: trmnl-renderer-secrets
+                  key: BYOS_ACCESS_TOKEN
+            - name: BYOS_REFRESH_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: trmnl-renderer-secrets
+                  key: BYOS_REFRESH_TOKEN
+          volumeMounts:
+            - name: data
+              mountPath: /data
+            - name: schedule-template
+              mountPath: /template
+              readOnly: true
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+      containers:
+        # Pinned, not enrolled in image automation — same reasoning as
+        # Terminus itself. This drives a wall-mounted panel and upstream
+        # is pre-1.0; an unattended bump that changes the dithering
+        # pipeline or the webhook payload shows up as a corrupted
+        # screen, not as a failed rollout.
+        #
+        # Note the image is ARCH-SUFFIXED: upstream publishes no
+        # multi-arch manifest, only trmnl-ha-amd64 and trmnl-ha-aarch64.
+        - name: renderer
+          image: ghcr.io/usetrmnl/trmnl-ha-amd64:0.9.5
+          ports:
+            - name: http
+              containerPort: 10000
+          env:
+            # The in-cluster Service name, NOT ha.renner.uno and not
+            # home.renner.uno. Both public names are unusable from a
+            # pod: ha.renner.uno resolves to the home-assistant-bgp
+            # LoadBalancer's external address (10.2.99.87), which is not
+            # backed by an Ingress, so CoreDNS's k8s_gateway delegation
+            # does not intercept it and the pod hairpins off the LB.
+            #
+            # This cluster has hit that exact failure before — see the
+            # comment on kubernetes/infrastructure/openbao/rotation/
+            # home-assistant.yaml, which reaches HA the same way and
+            # states it outright: "The public name is unusable from a
+            # pod (ingress-LB hairpin)."
+            #
+            # FAILS QUIETLY if wrong: the schedule keeps firing every
+            # minute, each render errors on connect, and the panel just
+            # never updates. Nothing goes CrashLoopBackOff.
+            - name: HOME_ASSISTANT_URL
+              value: "http://home-assistant-bgp.home-assistant.svc.cluster.local:8123"
+            - name: TZ
+              value: "Europe/Berlin"
+            # Chromium will not start without a writable HOME. Running
+            # as uid 1000 with no matching passwd entry leaves HOME="/",
+            # which is root-owned 0755, and the browser dies on launch
+            # with
+            #   chrome_crashpad_handler: --database is required
+            # because crashpad cannot create its database directory.
+            #
+            # Diagnosed on the running pod, not guessed: HOME=/, uid=1000,
+            # / not writable, /tmp writable. Crash dumps are throwaway, so
+            # /tmp is the right home for them rather than the PVC.
+            #
+            # This is a consequence of OUR non-root securityContext, not
+            # of the image — so it must move together with runAsUser.
+            - name: HOME
+              value: "/tmp"
+            # Dedicated long-lived token, NOT the operator's personal
+            # one, so the renderer's access can be revoked on its own.
+            # It is injected into the headless browser's localStorage,
+            # which is how the app authenticates to the HA frontend.
+            - name: ACCESS_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: trmnl-renderer-secrets
+                  key: HA_ACCESS_TOKEN
+          volumeMounts:
+            - name: data
+              mountPath: /data
+            # Bounded scratch backing HOME. Chromium writes crashpad
+            # state and profile data here on every launch, and it now
+            # launches once a minute for the life of the pod — on the
+            # container's writable layer that is unbounded growth
+            # against node disk, with no ephemeral-storage limit to
+            # catch it. A sizeLimit converts a slow, cluster-wide
+            # node-disk-pressure problem into a bounded and clearly
+            # attributable eviction of this pod.
+            - name: tmp
+              mountPath: /tmp
+          # HTTP probes here, unlike terminus.yaml's TCP-only ones, because
+          # /health is confirmed rather than assumed: the upstream image's
+          # own Dockerfile declares
+          #   HEALTHCHECK ... curl -f http://localhost:10000/health
+          # so the route is part of the image's published contract. Terminus
+          # got TCP probes precisely because it publishes no such endpoint.
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 250m
+              memory: 768Mi
+            # Chromium is a SEPARATE process from the app, and the app's
+            # own watchdog only measures its own RSS (restarting itself
+            # past 900 MB). The limit therefore has to cover the app's
+            # ceiling PLUS a browser rendering a 1872x1404 page, which
+            # is well above the 512 MB upstream's troubleshooting note
+            # suggests.
+            limits:
+              memory: 2Gi
+          # Chromium normally needs either SYS_ADMIN or a user namespace to
+          # start its own sandbox, which would make dropping ALL fatal. It
+          # is safe here because the app launches the browser with
+          # --no-sandbox --disable-setuid-sandbox --no-zygote (and
+          # --disable-dev-shm-usage, which is why no /dev/shm emptyDir is
+          # mounted either). Those flags are baked into the app's own
+          # PUPPETEER_ARGS, not something we pass, so this holds as long as
+          # the pinned tag does — worth re-checking on any bump.
+          #
+          # Confirmed on the running pod: the capability drop is genuinely
+          # not what broke the browser. The first deploy DID crash it, but
+          # on a writable-HOME problem caused by runAsUser — see the HOME
+          # env above. Do not "fix" a future Chromium launch failure by
+          # loosening this block before checking HOME and /tmp.
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: trmnl-renderer-data
+        - name: schedule-template
+          configMap:
+            name: trmnl-renderer-schedule
+        - name: tmp
+          emptyDir:
+            sizeLimit: 512Mi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: trmnl-renderer
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: trmnl-renderer
+spec:
+  # ClusterIP only, and deliberately no Ingress. The app's web UI on
+  # this port has NO authentication of any kind — upstream says so
+  # outright — so it must not be routable from the LAN. Terminus
+  # reaches it in-namespace to fetch rendered screens; nothing else
+  # needs to.
+  selector:
+    app.kubernetes.io/name: trmnl-renderer
+  ports:
+    - name: http
+      port: 10000
+      targetPort: http

kubernetes/apps/trmnl/scheduled-backup.yaml

@@ -0,0 +1,16 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: tr-pg-daily
+  namespace: trmnl
+spec:
+  # 03:00 local, same slot as the other CNPG clusters (the Velero PVC
+  # schedule for this namespace runs later, at 08:15).
+  schedule: "0 0 3 * * *"
+  immediate: true
+  backupOwnerReference: self
+  cluster:
+    name: tr-pg
+  method: plugin
+  pluginConfiguration:
+    name: barman-cloud.cloudnative-pg.io

kubernetes/apps/trmnl/terminus.yaml

@@ -0,0 +1,256 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: terminus-config
+  namespace: trmnl
+data:
+  TZ: "Europe/Berlin"
+  HANAMI_PORT: "2345"
+  # What the device is told to fetch from. Terminus bakes this into the
+  # absolute `image_url` it hands back on /api/display, so it MUST be the
+  # cleartext LAN name, not trmnl.renner.uno — the firmware speaks plain
+  # HTTP and would reject the internal CA. No trailing slash: the same
+  # string is typed into the captive portal at provisioning time.
+  API_URI: "http://trmnl-device.renner.uno"
+  KEYVALUE_URL: "redis://terminus-valkey:6379"
+  # Rack::Attack rate-limit bypass. Beyond upstream's RFC1918 defaults this
+  # adds fc00::/7, which covers the pod CIDR (fd10:244::/48), the Cilium LB
+  # pool (fd99::/120) and the LAN ULA — without it every IPv6 client,
+  # including the device and Traefik, is rate-limited as if it were remote.
+  RACK_ATTACK_ALLOWED_SUBNETS: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1,::1,fc00::/7"
+  # 10 years, against a default of 1800s (30 minutes).
+  #
+  # Documented upstream in doc/configuration.adoc:
+  #   API_ACCESS_TOKEN_PERIOD — "The amount of time, in seconds, the JWT
+  #   access token remains valid before needing to refresh your access
+  #   token." Default: 1,800 seconds (30 minutes).
+  # Corroborated on this instance: a token minted here was rejected about
+  # thirty minutes later, i.e. the documented default behaving as written.
+  #
+  # This exists for the renderer, which pushes screens to /api/screens with
+  # a JWT held in OpenBao. At the default period that arrangement cannot
+  # work unattended, and it fails in a way that has no recovery: minting a
+  # replacement token via POST /api/jwt requires a still-VALID access token
+  # in the Authorization header, so once one expires it cannot be refreshed
+  # from itself. Observed exactly that on first deploy —
+  #   [webhook] Webhook failed: 400 {"error":"expired JWT access token"}
+  #   [webhook] BYOS auth: refresh failed (400) {"error":"expired JWT ..."}
+  # — the refresh attempt being rejected by the very expiry it was trying
+  # to repair.
+  #
+  # The renderer's own code assumes "stock Terminus disables session
+  # expiration and issues effectively permanent tokens". That is simply not
+  # true of this deployment, so the assumption is made true here rather
+  # than worked around in the client.
+  #
+  # The trade is deliberate: a long-lived JWT with full Terminus API rights
+  # now sits in OpenBao. Accepted because the alternative was putting the
+  # Terminus account PASSWORD into the cluster so an init container could
+  # mint a short-lived token on every pod start, which is a strictly worse
+  # secret to hold. The token is revocable by rotating the account's
+  # credentials; it grants access only to this LAN-only instance.
+  #
+  # NOTE: the period is fixed at mint time. Changing this does not extend
+  # tokens that already exist — they must be re-minted afterwards.
+  API_ACCESS_TOKEN_PERIOD: "315360000"
+  # The OTHER half of the same problem, and the reason the first fix
+  # did not hold. Terminus has TWO independent expiries: the JWT's own
+  # lifetime (above) and the SESSION behind it. Raising only the token
+  # period left the session at its defaults, so pushes kept working for
+  # a while and then failed with
+  #   HTTP 401 {"error":"This session has expired, please login again"}
+  # which is a different error, from a different mechanism, than the
+  # "expired JWT access token" the first fix addressed.
+  #
+  # Documented in doc/configuration.adoc:
+  #   SESSION_EXPIRATION_ENABLED — "Determines if your browser session
+  #   or JWT API token is expired or not." Default: true.
+  #   SESSION_INACTIVITY_LIMIT 1800s, SESSION_LIFETIME_LIMIT 86400s.
+  #
+  # Disabling it is exactly the state the renderer's own code assumes
+  # ("stock Terminus disables session expiration and issues ~100-year
+  # tokens") — an assumption that is false by default and has to be
+  # made true here, same as the token period.
+  #
+  # Same trade as above, and it is the deciding one for an unattended
+  # appliance: a session that expires needs a human with a password to
+  # restore it, and this panel has neither.
+  SESSION_EXPIRATION_ENABLED: "false"
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: terminus
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus
+  annotations:
+    # terminus-config is consumed via envFrom, and env vars are read ONCE at
+    # container start — Kubernetes does not hot-reload them. The ConfigMap is
+    # a plain resource, not a configMapGenerator, so editing it does not
+    # change the pod template hash and Flux will not roll the pods either.
+    # Without this annotation a ConfigMap-only change (such as raising
+    # API_ACCESS_TOKEN_PERIOD) applies to git and to the ConfigMap object
+    # while the running app keeps the old value indefinitely — a silent
+    # no-op, with the manifest and reality disagreeing.
+    #
+    # Named per-ConfigMap rather than the blanket `reloader.stakater.com/
+    # auto: "true"`, matching mosquitto-deployment.yaml: enrolling anything
+    # further stays a visible, reviewable edit to this line.
+    configmap.reloader.stakater.com/reload: terminus-config
+spec:
+  replicas: 1
+  # RWO PVC shared by both containers in this pod, and APP_SETUP runs
+  # migrations on boot — two concurrent web pods would race on both.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: terminus
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: terminus
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        # Tag pinned deliberately and NOT enrolled in Flux image automation.
+        # Terminus is pre-1.0 with no GitHub Releases and no changelog; an
+        # unattended bump that changes the firmware or /api/display contract
+        # is a device-bricking class of risk. Bumps are a reviewed, manual
+        # act — see kubernetes/infrastructure/image-automation/README.md.
+        - name: terminus
+          image: ghcr.io/usetrmnl/terminus:0.68.0
+          ports:
+            - name: http
+              containerPort: 2345
+          envFrom:
+            - configMapRef:
+                name: terminus-config
+          env:
+            # Runs `hanami assets compile` + `hanami db migrate` in the
+            # entrypoint before exec'ing puma. Set on the web container
+            # only — the sidekiq sidecar must not race the migration.
+            - name: APP_SETUP
+              value: "true"
+            # CNPG generates this Secret; `uri` already carries the rotated
+            # password and points at the tr-pg-rw service.
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: tr-pg-app
+                  key: uri
+            - name: APP_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: terminus-secrets
+                  key: APP_SECRET
+          volumeMounts:
+            - name: data
+              mountPath: /app/public/uploads
+              subPath: uploads
+            - name: data
+              mountPath: /app/public/fonts
+              subPath: fonts
+          # TCP probes, not HTTP: Terminus exposes no documented health
+          # endpoint, and every real route is behind either auth or
+          # Rack::Attack. Tighten to an HTTP GET once the route set is
+          # confirmed against the running app.
+          startupProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 10
+            # ~10 min: first boot compiles assets and migrates a fresh DB.
+            failureThreshold: 60
+          livenessProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 10
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              memory: 1Gi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+        - name: sidekiq
+          image: ghcr.io/usetrmnl/terminus:0.68.0
+          command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
+          envFrom:
+            - configMapRef:
+                name: terminus-config
+          # EXPECTED on a fresh deploy: this container loses the race against
+          # the web container's migration and exits 1 with
+          # `PG::UndefinedTable: relation "account" does not exist`, then
+          # comes up healthy on the restart once the schema exists.
+          # Observed on first deploy 2026-08-10; converged in ~1s. There is
+          # no upstream ordering hook, and gating it behind an initContainer
+          # that probes for a specific table would couple us to Terminus's
+          # schema. So this is left to Kubernetes' restart, deliberately —
+          # a single sidekiq restart shortly after a rollout is benign. More
+          # than a couple, or restarts long after a rollout, are NOT this.
+          env:
+            # No APP_SETUP here — see the web container.
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: tr-pg-app
+                  key: uri
+            - name: APP_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: terminus-secrets
+                  key: APP_SECRET
+          volumeMounts:
+            - name: data
+              mountPath: /app/public/uploads
+              subPath: uploads
+            - name: data
+              mountPath: /app/public/fonts
+              subPath: fonts
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 512Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: terminus-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: terminus
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus
+spec:
+  selector:
+    app.kubernetes.io/name: terminus
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

kubernetes/apps/trmnl/valkey.yaml

@@ -0,0 +1,79 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: terminus-valkey
+  namespace: trmnl
+  labels:
+    app.kubernetes.io/name: terminus-valkey
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: terminus-valkey
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: terminus-valkey
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 999
+        fsGroup: 999
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        # Valkey major version tracks upstream's compose.yml (9.x). Sidekiq
+        # is the only consumer; a major bump is a separate migration.
+        - name: valkey
+          image: valkey/valkey:9.1.1-alpine # {"$imagepolicy": "flux-system:valkey-trmnl"}
+          args:
+            - valkey-server
+            - --save
+            - "60"
+            - "1"
+            - --loglevel
+            - warning
+          ports:
+            - name: valkey
+              containerPort: 6379
+          volumeMounts:
+            - name: data
+              mountPath: /data
+          livenessProbe:
+            tcpSocket:
+              port: valkey
+            initialDelaySeconds: 10
+          readinessProbe:
+            exec:
+              command: ["valkey-cli", "ping"]
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 20m
+              memory: 64Mi
+            limits:
+              memory: 256Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: terminus-valkey-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: terminus-valkey
+  namespace: trmnl
+spec:
+  selector:
+    app.kubernetes.io/name: terminus-valkey
+  ports:
+    - port: 6379
+      targetPort: valkey

kubernetes/apps/whisper/deployment.yaml

@@ -18,7 +18,7 @@ spec:
     spec:
       containers:
         - name: whisper
-          image: rhasspy/wyoming-whisper:3.5.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
+          image: rhasspy/wyoming-whisper:3.6.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
           args:
             - --model=base-int8
             - --language=de

kubernetes/apps/woodpecker/MIGRATION-namespace.md

@@ -0,0 +1,388 @@
+# Woodpecker: `apps` → `woodpecker` namespace migration
+
+Moves the Woodpecker service — server, agent and the `wp-pg` CNPG backend — out
+of the shared `apps` namespace into one of its own. Distinct from
+`MIGRATION.md`, which covers the earlier SQLite → CNPG move.
+
+## Why
+
+`apps` holds gitea, gatus, external-dns, ical-proxy and woodpecker together,
+and its `default-deny-ingress` uses `endpointSelector: {}`. Every allow rule it
+carries therefore applies to *every* workload in the namespace.
+
+That is not academic. Issue #433 — `wp-pg` stuck at `Ready=False` with
+`Instance Status Extraction Error` for three weeks — is caused by the CNPG
+operator in `database` being unable to reach `https://<pod-ip>:8000/pg/status`.
+The only namespace-level fix is `fromEndpoints: namespace=database`, which
+would expose gitea, gatus, external-dns and ical-proxy to the whole `database`
+namespace to solve a problem affecting one pod on one port.
+
+`cnpg_collector_up` has series for authentik, immich, monitoring and paperless
+but not for `wp-pg`, for the same reason: `apps` never allowed the monitoring
+namespace either.
+
+A namespace per service keeps each grant on the workload that needs it.
+
+**`woodpecker-ci` is not the target.** It enforces `pod-security: privileged`
+and exists solely for ephemeral pipeline step pods running repo-supplied code.
+Putting the database there would let any build step reach it through the
+intra-namespace allow.
+
+## Staging
+
+Rollback is trivial through stage 2. The `apps` cluster is untouched until
+stage 3, so reverting means pointing the DSN back and scaling up.
+
+| Stage | Contents | CI impact |
+|---|---|---|
+| **1** (this PR) | `woodpecker` namespace, its network policies, ExternalSecret, ObjectStores, and `wp-pg` restored from barman | none — parallel, nothing points at it |
+| **runbook** | verify the restore holds real data | none |
+| **2** | move server + agent, flip DSN, RBAC subject, rotator policies, docs | **downtime** |
+| **3** | delete the `apps` originals, add the prune guard to the survivor | none |
+
+## Stage 1 design notes
+
+**Two ObjectStores, deliberately.** `wp-pg-store-source` points at
+`s3://renner-uno-k8s-cnpg/wp-pg` — the prefix the live cluster archives to —
+and is read-only. `wp-pg-store` points at a **new** prefix,
+`s3://renner-uno-k8s-cnpg/wp-pg-woodpecker`, for ongoing archiving.
+
+Both clusters writing one `destinationPath` would interleave their WAL streams
+and corrupt both. Splitting them also leaves the pre-migration backup intact
+and independently restorable.
+
+**`serverName: wp-pg`** on the `externalCluster` names the cluster that
+*produced* the backup, not the one being created.
+
+**The namespace is declared in `infrastructure/namespaces/namespaces.yaml`,**
+not alongside these workloads. Its CiliumNetworkPolicies ship with the
+`infrastructure-crds` KS (`infrastructure/post-crds` includes
+`../network-policies/`), which `dependsOn` `infrastructure` — so the namespace
+exists before any policy targets it. Creating it in the `woodpecker` workload
+KS would invert that order, since that KS runs after both.
+
+**Ingress from `woodpecker-ci` is deliberately absent in stage 1.** Step pods do
+call the server back on `:8000`, but the CNPG instance manager serves its status
+API on `:8000` too, and the baseline policy uses `endpointSelector: {}`. A
+`toPorts` block narrows the port, not the destination workload — so putting that
+rule on the baseline would hand arbitrary repo-supplied build steps a route to
+wp-pg's instance manager. It lands in stage 2 as its own policy selecting the
+server pod.
+
+**No `prune: disabled` guard on the restored cluster.** The original carries one
+because it holds the only copy of the data. This one is reconstructable by
+re-running the restore, and a prune guard on a migration artifact is how orphans
+survive an abandoned migration. It gains the guard in stage 3.
+
+**Barman credentials still come from `seed/paperless/hetzner-creds`.** Issue
+#397 wants woodpecker to own `seed/woodpecker/hetzner-creds`, and this would
+have been the natural moment — but creating that path needs OpenBao write
+access that was not available (`bao kv list seed` → 403). Repointing at a path
+that may not exist would leave the restored cluster unable to archive WAL,
+which is the thing this stage exists to prove. #397 stays open; the swap is one
+line once the path exists.
+
+## Expect the `woodpecker` Kustomization to go not-Ready during the restore
+
+`kubernetes/clusters/homelab/woodpecker.yaml` sets `wait: true` and a
+`healthCheckExprs` entry matching `postgresql.cnpg.io/v1 Cluster` **with no
+name or namespace qualifier**. Flux evaluates it against every Cluster in that
+Kustomization's inventory — which, after this stage, is both the live
+`apps/wp-pg` and the restoring `woodpecker/wp-pg`.
+
+So while the restore runs, the whole `woodpecker` KS — the one that owns
+production Woodpecker — reports not-Ready, and a reconcile timeout in that
+window will trip the `flux-failures` Alert. **That is expected and is not
+Woodpecker breaking.** Nothing `dependsOn: woodpecker`, so it does not cascade.
+
+If the restore is going to sit for a long time, or the noise is unwelcome, the
+options are to suspend the KS while it completes, or to split the migration
+resources into their own `wait: false` Kustomization. For a ~660 MB database the
+window should be minutes.
+
+## Verifying stage 1
+
+Restore is asynchronous. Watch it reach a healthy state:
+
+```sh
+kubectl -n woodpecker get cluster wp-pg -w
+kubectl -n woodpecker get pods
+```
+
+Expect `Cluster in healthy state`. If it sticks in bootstrap, the recovery job's
+logs carry the reason:
+
+```sh
+kubectl -n woodpecker logs -l cnpg.io/jobRole=full-recovery --tail=100
+```
+
+**First check the generated app credentials target the right database.** CNPG
+defaults `bootstrap.recovery.database` to `app`; the restored data uses
+`woodpecker`. If this reads anything but `woodpecker`, the cluster has
+provisioned an empty `app` database and the real data is unreachable through
+the `-app` Secret and Service:
+
+```sh
+kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.dbname}' | base64 -d; echo
+kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.username}' | base64 -d; echo
+```
+
+**Then confirm the restore actually holds Woodpecker's data** — a cluster that
+comes up healthy but empty is the failure mode worth catching:
+
+```sh
+PGPASS=$(kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
+kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
+  psql -U woodpecker -d woodpecker -c '\dt'
+kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
+  psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
+```
+
+Row counts should match the live cluster:
+
+```sh
+PGPASS_OLD=$(kubectl -n apps get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
+kubectl -n apps exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS_OLD" \
+  psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
+```
+
+They will not match exactly if builds ran after the backup the restore used.
+
+**And stage 2 does not close that gap.** An earlier draft of this document
+claimed it quiesced the server first; it does not — Flux simply prunes the
+`apps` workloads and creates the `woodpecker` ones, so anything written to the
+old database between the restore and the cutover stays only in the old database.
+
+Measured on the live cutover: the restore captured through pipeline `9488`,
+four more (`9491`, `9492`, `9494`, `9496`) landed in `apps/wp-pg` before the
+swap, and the new cluster resumed at `9521`. Four pipeline records and their
+log entries exist only in the old cluster. No ID collision, no functional
+breakage — Woodpecker's own manifest calls this database "small and
+non-critical (state is largely reconstructable)" — but it is real history loss
+and should be a decision, not a surprise.
+
+If that gap matters for a future migration, quiesce before cutting over:
+
+```sh
+flux suspend kustomization woodpecker
+kubectl -n <old-ns> scale deploy/woodpecker-server deploy/woodpecker-agent --replicas=0
+# re-run the restore so it captures the final state, then resume
+```
+
+The old cluster keeps its `prune: disabled` guard until stage 3, so the missing
+rows remain recoverable until then.
+
+Confirm the two fixes this move exists for:
+
+```sh
+# #433: operator can now reach the instance
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+# metrics now scraped — needs BOTH the PodMonitor (cnpg-podmonitors.yaml) and
+# the monitoring ingress allow; wp-pg had neither in `apps`, which is why it is
+# absent from cnpg_collector_up while ak-pg/im-pg/gf-pg/pprl-pg are present
+kubectl get --raw '/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=cnpg_collector_up{namespace="woodpecker"}'
+```
+
+Confirm the scrape target is actually registered, not just that the metric is
+missing for some other reason:
+
+```sh
+kubectl -n woodpecker get podmonitor cnpg-wp-pg
+```
+
+**Check the policy semantics by hand.** `kubectl --dry-run=server` validates
+manifest shape, not Cilium behaviour, so it cannot catch a rule that is
+syntactically fine but scoped wider than intended.
+
+`cilium-dbg policy trace` does **not** exist in Cilium 1.20 — it was removed,
+and `cilium-dbg policy` now offers only `get`, `selectors`, `subject-selectors`
+and `wait`. Read the realized BPF policy map instead, which is better evidence
+anyway: it is the datapath as programmed, not a simulation of it, and it
+carries live byte/packet counters.
+
+Find the agent on wp-pg's node and the endpoint id:
+
+```sh
+kubectl -n woodpecker get pod wp-pg-1 -o jsonpath='{.spec.nodeName}'
+kubectl -n kube-system get pod -l k8s-app=cilium \
+  --field-selector spec.nodeName=<node> --no-headers -o custom-columns=N:.metadata.name
+
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg endpoint list -o json \
+  | jq -r '.[] | select((.status.labels."security-relevant"[]?|tostring)
+      | test("cnpg.io/cluster=wp-pg")) | "\(.id) \(.status.networking.addressing[0].ipv4)"'
+```
+
+Then, on that endpoint id — who may reach `:8000`:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | awk '$4 ~ /8000/ || NR<=1'
+```
+
+Expect exactly one row, the CNPG operator, with non-zero counters once it has
+polled:
+
+```
+Allow  Ingress  k8s:app.kubernetes.io/instance=cloudnative-pg  8000/TCP  11756  58
+```
+
+And confirm CI step pods reach **ICMP only** — never a TCP port:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | grep woodpecker-ci | awk '{print $4}' | sort -u
+```
+
+Every line must be an ICMP/ICMPv6 type. A bare port number here means the
+namespace baseline has grown a rule it should not have.
+
+Note the operator's allow is also confirmed indirectly and more strongly by
+`Ready=True` on the Cluster: that condition only goes true once the operator has
+successfully polled this exact endpoint and port.
+
+And that archiving goes to the new prefix, leaving the old one alone:
+
+```sh
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")]}'
+```
+
+## Stage 2 — the cutover
+
+Moves server, agent, service, ingress, pdb, the agent ServiceAccount, the
+`woodpecker-secret` ExternalSecret and the rotator policy into `woodpecker`,
+points the server's DSN at `wp-pg-rw.woodpecker.svc`, repoints the agent's gRPC
+at `woodpecker.woodpecker.svc:9000`, moves the openbao rotators' egress, and
+adds the `woodpecker-ci` → server `:8000` policy that stage 1 deliberately
+deferred — scoped to the server pod, not the namespace baseline.
+
+**This is the downtime window.** Flux prunes the `apps` copies and creates the
+`woodpecker` ones; `ci.renner.uno` is down in between, and in-flight pipelines
+die with their agent.
+
+**The outage outlasts the pods becoming Ready, because of TLS.** Certificates
+are namespace-scoped, so moving the Ingress does not move `woodpecker-tls` —
+cert-manager issues a fresh one in the new namespace. Until it completes,
+Traefik serves a default certificate and clients fail before they get an HTTP
+status at all:
+
+```
+curl: (60) SSL: no alternative certificate subject name matches target hostname 'ci.renner.uno'
+```
+
+That is expected, not a routing or policy fault. Watch it clear with:
+
+```sh
+kubectl -n woodpecker get certificate woodpecker-tls -w
+```
+
+`letsencrypt-cluster` solves **DNS-01** via Cloudflare, so there is no in-cluster
+ACME solver pod and no HTTP challenge to admit through the network policies —
+worth knowing before scoping any ingress rule, since an HTTP-01 issuer would
+need the solver reachable too.
+
+The four CNPG files (`cluster.yaml`, `objectstore.yaml`, `scheduled-backup.yaml`,
+`hetzner-creds-externalsecret.yaml`) still describe the **original** apps-namespace
+cluster and are deliberately untouched — they are the rollback. Stage 3 retires
+them.
+
+### Verifying stage 2
+
+```sh
+kubectl -n woodpecker get pods
+kubectl -n apps get pods -l app.kubernetes.io/name=woodpecker   # expect none
+```
+
+The server must reach the restored database, not the old one:
+
+```sh
+kubectl -n woodpecker logs deploy/woodpecker-server --tail=40 | grep -i "database\|migrat\|error"
+kubectl -n woodpecker exec deploy/woodpecker-server -- env | grep DATASOURCE
+```
+
+Then confirm the UI and a real pipeline, which is the only check that exercises
+server → agent → step pod → server end to end:
+
+```sh
+curl -sS -o /dev/null -w '%{http_code}\n' https://ci.renner.uno/
+woodpecker pipeline create --branch master --var PIPELINE=e2e-test k.renner/homelab
+```
+
+Step pods must still reach the server on `:8000` while still being unable to
+reach wp-pg on any TCP port — re-run the `bpf policy get` checks above against
+both the server's and wp-pg's endpoint ids.
+
+Check the rotators can still reach the server, since their egress moved:
+
+```sh
+kubectl -n openbao get cronjob
+kubectl -n openbao logs -l app.kubernetes.io/component=secret-rotator --tail=20
+```
+
+### Rolling back stage 2
+
+Revert the PR. Flux recreates the `apps` workloads and the DSN points back at
+`wp-pg-rw.apps.svc`, which never stopped running. Data written to the new
+cluster during the window is lost, so roll back promptly or not at all — if
+pipelines have run against `woodpecker/wp-pg`, prefer fixing forward.
+
+## Stage 3 — retiring the old cluster
+
+Deletes the four `apps` manifests, promotes the `migration-*` set onto their
+names, and gives the surviving Cluster the `prune: disabled` data guard it was
+deliberately denied while a rollback still existed.
+
+**Merging stage 3 does not delete anything.** The old resources carry
+`kustomize.toolkit.fluxcd.io/prune: disabled`, so dropping them from git only
+stops Flux managing them — they keep running, and keep occupying their PVC.
+That is the point: the PR is reversible, the teardown is a separate, deliberate
+act.
+
+Before deleting anything, confirm the new cluster is genuinely standing on its
+own — that it has taken at least one base backup to its own prefix, so recovery
+no longer depends on the retired cluster's data:
+
+```sh
+kubectl -n woodpecker get backup.postgresql.cnpg.io
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+```
+
+`LastBackupSucceeded` must be `True`, not merely `BackupStarted`.
+
+Then retire the old set:
+
+```sh
+kubectl -n apps delete scheduledbackup.postgresql.cnpg.io wp-pg-daily
+kubectl -n apps delete cluster.postgresql.cnpg.io wp-pg          # deletes its PVCs
+kubectl -n apps delete objectstore.barmancloud.cnpg.io wp-pg-store
+kubectl -n apps delete externalsecret hetzner-s3-creds
+```
+
+Delete the ScheduledBackup first: leaving it while the Cluster goes away
+produces failing backup jobs against a cluster that no longer exists.
+
+**What this destroys and what it does not.** It destroys the old cluster's PVCs
+— including the four pipeline records that only ever existed there (see the
+stage-1 verification section). It does **not** touch
+`s3://renner-uno-k8s-cnpg/wp-pg`: that prefix keeps its backups until their own
+retention expires, and `wp-pg-store-source` is retained precisely so it stays
+restorable. That is the real safety net, not the running cluster.
+
+### After the teardown
+
+The `apps` namespace should have no woodpecker-related objects left:
+
+```sh
+kubectl -n apps get cluster,objectstore,scheduledbackup,externalsecret 2>/dev/null | grep -i "wp-pg\|hetzner" || echo "clean"
+```
+
+## Rolling back stage 1
+
+Delete the four resources and the namespace. Nothing references them, and the
+`apps` cluster is untouched:
+
+```sh
+kubectl delete ns woodpecker
+```
+
+Then revert this PR so Flux stops recreating them.

kubernetes/apps/woodpecker/agent.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -12,7 +12,7 @@ metadata:
     secret.reloader.stakater.com/reload: woodpecker-secret
 spec:
   # 1: the in-cluster server owns ci.renner.uno now, so this agent registers with
-  # it (woodpecker.apps.svc:9000) and runs pipelines on the kubernetes backend,
+  # it (woodpecker.woodpecker.svc:9000) and runs pipelines on the kubernetes backend,
   # replacing the retired docker agents (LXC 910/911).
   replicas: 1
   selector:
@@ -37,7 +37,7 @@ spec:
           image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
           env:
             - name: WOODPECKER_SERVER
-              value: woodpecker.apps.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
+              value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
             - name: WOODPECKER_MAX_WORKFLOWS
               value: "2"
             # --- kubernetes backend: pipeline steps run as native pods (retires

kubernetes/apps/woodpecker/cluster.yaml

@@ -1,19 +1,27 @@
+---
+# Woodpecker's Postgres backend. Serves the server at wp-pg-rw.woodpecker.svc.
+#
+# Originally created in `apps` with bootstrap.initdb; moved here by restoring
+# from barman into the new namespace rather than by editing a namespace field,
+# because PVCs are namespace-scoped and a Cluster cannot be relocated in place
+# (issue #433, MIGRATION-namespace.md).
+#
+# The bootstrap.recovery stanza and its externalCluster are retained rather than
+# reverted to initdb. CNPG honours `bootstrap` only when PGDATA is absent, so on
+# this live cluster it is inert — and keeping it means a cold rebuild of an
+# empty cluster restores from backup instead of coming up empty, which is the
+# gap kubernetes/RECOVERY.md exists to close for the other CNPG clusters.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
   name: wp-pg
-  namespace: apps
+  namespace: woodpecker
   annotations:
-    # CRITICAL data-guard: prune-disabled so the seed-woodpecker -> dedicated
-    # woodpecker KS ownership handoff can never prune this Cluster (which would
-    # delete its data PVCs). Landed here first, via seed-woodpecker, so it's
-    # live before the move. Kept permanently as a standing guard.
+    # Data guard, earned at stage 3 now that this is the only copy: prevents a
+    # Flux prune from deleting the Cluster and with it the data PVCs. The
+    # migration deliberately withheld this while a rollback still existed.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # Single instance: the Woodpecker CI DB is small and non-critical (state is
-  # largely reconstructable), so HA isn't warranted — but WAL archiving to
-  # object storage still gives point-in-time backups. Mirrors grafana-db's
-  # single-instance shape, with barman enabled like immich/paperless.
   instances: 1
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
@@ -25,9 +33,25 @@ spec:
       parameters:
         barmanObjectName: wp-pg-store
   bootstrap:
-    initdb:
+    recovery:
+      source: wp-pg-apps
+      # Must mirror the initdb values of the cluster this restores from
+      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
+      # `database` to `app` and `owner` to whatever `database` is, and those
+      # names do not exist in the physically restored data — it would provision
+      # an empty `app` database, point the generated wp-pg-app Secret and the
+      # -app Service at it, and leave the real data unreachable through them.
+      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
       database: woodpecker
       owner: woodpecker
+  externalClusters:
+    - name: wp-pg-apps
+      plugin:
+        name: barman-cloud.cloudnative-pg.io
+        parameters:
+          barmanObjectName: wp-pg-store-source
+          # The cluster that produced the backup, not this one.
+          serverName: wp-pg
   postgresql:
     parameters:
       shared_buffers: "128MB"

kubernetes/apps/woodpecker/externalsecret.yaml

@@ -2,7 +2,7 @@ apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
 spec:
   # 10m (was 1h) so an automated rotation (rotate-agent-secret.yaml writes a new
   # value to OpenBao) propagates into the k8s Secret within ~10m without a manual

kubernetes/apps/woodpecker/hetzner-creds-externalsecret.yaml

@@ -1,11 +1,20 @@
+---
+# Barman S3 credentials for wp-pg.
+#
+# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
+# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
+# that path needs OpenBao write access (`bao kv list seed` → 403 for the
+# identities available here), so it stays open. The values are identical, so it
+# is a one-line swap of the `key:` below once the path exists.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace, or ESO silently never syncs — see #505.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: hetzner-s3-creds
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (barman S3 creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
   refreshInterval: 1h
   secretStoreRef:
@@ -14,10 +23,5 @@ spec:
   target:
     name: hetzner-s3-creds
   dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration. FOLLOW-UP: give woodpecker its own
-    # seed/woodpecker/hetzner-creds path once OpenBao can be written, and repoint
-    # this here (the value is the same, so it's a drop-in swap).
     - extract:
         key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/ingress.yaml

@@ -2,7 +2,7 @@ apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
   annotations:

kubernetes/apps/woodpecker/kustomization.yaml

@@ -21,11 +21,20 @@ resources:
   # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml) — intra-ns
   # OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
   # Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
-  # wp-pg CNPG Postgres backend (moved here from the retired seed-woodpecker KS;
-  # each object carries prune: disabled so ownership transferred without a
-  # teardown). The dedicated `woodpecker` KS depends on infrastructure-cnpg-barman
-  # for the barman plugin behind the ObjectStore/ScheduledBackup.
-  - cluster.yaml
-  - objectstore.yaml
-  - scheduled-backup.yaml
+  # --- namespace migration, stages 1-2 (issue #433) ---
+  # Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
+  # rbac, secrets, policies) into the `woodpecker` namespace and pointed the
+  # server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
+  # originals; these four now describe the surviving cluster. History and the
+  # reasoning behind the three stages: MIGRATION-namespace.md.
+  #
+  # The `woodpecker` Namespace itself is declared in infrastructure/namespaces,
+  # not here: it must exist before the `infrastructure-crds` KS applies the
+  # namespace's CiliumNetworkPolicies, and this KS runs after both of those.
+  #
+  # This KS depends on infrastructure-cnpg-barman for the barman plugin behind
+  # the ObjectStore and ScheduledBackup.
   - hetzner-creds-externalsecret.yaml
+  - objectstore.yaml
+  - cluster.yaml
+  - scheduled-backup.yaml

kubernetes/apps/woodpecker/networkpolicy.yaml

@@ -1,20 +1,23 @@
 # Secret rotation: the review-bot and claude-oauth rotators (openbao ns) reach
 # the Woodpecker SERVER to upsert org-level CI secrets (review_bot_token,
 # claude_code_oauth_token). They use the in-cluster Service
-# (woodpecker.apps.svc:8000) — the public ci.renner.uno name hairpins off the
-# ingress LB from a pod. The apps default-deny-ingress only allows same-ns /
-# host / a few named sources, so without this the rotator's connect to :8000 is
+# (woodpecker.woodpecker.svc:8000) — the public ci.renner.uno name hairpins off the
+# ingress LB from a pod. This namespace's baseline
+# (network-policies/woodpecker-ns.yaml) admits only same-namespace, monitoring,
+# host/remote-node and ICMP, so without this the rotator's connect to :8000 is
 # dropped. Scoped to the woodpecker SERVER pod :8000 and the two rotator labels;
 # the egress halves are rotator-review-bot-egress + rotator-claude-oauth-egress
 # in network-policies/openbao-ns.yaml.
 #
-# This is additive to the apps-ns baseline allows (allow-ingress.yaml selects all
-# apps pods), so the server's existing ingress — Traefik, agents — is unaffected.
+# Cilium unions allow rules, so this composes with the baseline and with the
+# sibling policies below (traefik, CI steps) rather than replacing them. Note
+# allow-ingress.yaml does NOT apply here — it is hard-scoped to `namespace:
+# apps`, which is why traefik needs its own rule in this file now.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
   name: woodpecker-allow-rotators
-  namespace: apps
+  namespace: woodpecker
 spec:
   endpointSelector:
     matchLabels:
@@ -34,3 +37,25 @@ spec:
         - ports:
             - port: "8000"
               protocol: TCP
+---
+# Pipeline step pods call the server back on :8000 to stream logs and report
+# status. Deferred from stage 1 on purpose, and scoped to the SERVER pod rather
+# than granted on the namespace baseline.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: woodpecker-allow-ci-steps
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: woodpecker
+      app.kubernetes.io/component: server
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": woodpecker-ci
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP

kubernetes/apps/woodpecker/objectstore.yaml

@@ -1,18 +1,20 @@
 ---
+# RECOVERY SOURCE — read-only. The prefix the retired `apps/wp-pg` archived to.
+#
+# Kept after the migration on purpose. Nothing writes to it, and the backups
+# under it age out on their own retention, but while they last this is a second,
+# independent restore point that predates the move — worth having until the new
+# prefix has a meaningful backup history of its own.
+#
+# serverName on the consuming externalCluster must be `wp-pg`, the name of the
+# cluster that produced these backups.
 apiVersion: barmancloud.cnpg.io/v1
 kind: ObjectStore
 metadata:
-  name: wp-pg-store
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup target).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  name: wp-pg-store-source
+  namespace: woodpecker
 spec:
   configuration:
-    # Same offsite Hetzner Object Storage bucket as the other CNPG clusters,
-    # under a per-cluster prefix. See apps/paperless/objectstore.yaml for the
-    # SSE / addressing rationale (Hetzner is SSE-C only; transit is TLS,
-    # bucket private).
     destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
     endpointURL: https://fsn1.your-objectstorage.com
     s3Credentials:
@@ -29,4 +31,36 @@ spec:
       compression: gzip
     data:
       compression: gzip
+  # No retentionPolicy: this store is a read source during the migration, and
+  # the old cluster in `apps` owns the lifecycle of the data underneath it.
+---
+# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
+#
+# Two clusters archiving WAL to one destinationPath interleave their streams and
+# corrupt both. That mattered during the parallel-run window; it still matters
+# now, because the source prefix above is retained and must stay frozen for the
+# independent restore point it provides.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: wp-pg-store
+  namespace: woodpecker
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
   retentionPolicy: "30d"

kubernetes/apps/woodpecker/pdb.yaml

@@ -15,7 +15,7 @@ apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server

kubernetes/apps/woodpecker/rbac.yaml

@@ -17,7 +17,7 @@ apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -60,4 +60,7 @@ roleRef:
 subjects:
   - kind: ServiceAccount
     name: woodpecker-agent
-    namespace: apps
+    # The agent's SA lives in `woodpecker`; this Role/RoleBinding stays in
+    # `woodpecker-ci`, because that is the namespace it grants pod
+    # create/delete in. Cross-namespace subject, deliberately.
+    namespace: woodpecker

kubernetes/apps/woodpecker/scheduled-backup.yaml

@@ -1,18 +1,19 @@
 ---
-# Daily base backup at 02:35 UTC (offset from the other clusters' 02:30 to
-# avoid a thundering herd against the Hetzner bucket). WAL archiving runs
-# continuously (see cluster.yaml spec.plugins[].isWALArchiver). immediate:true
-# takes the first base backup as soon as this CR applies.
+# Daily base backups onto the wp-pg-woodpecker prefix. Continuous WAL archiving
+# runs separately (cluster.yaml spec.plugins[].isWALArchiver); without periodic
+# base backups the prefix would have no recovery point of its own and
+# wp-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:40 UTC, offset from the other CNPG clusters' 02:30 so they do not all hit
+# the Hetzner bucket together.
 apiVersion: postgresql.cnpg.io/v1
 kind: ScheduledBackup
 metadata:
   name: wp-pg-daily
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup schedule).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
-  schedule: "0 35 2 * * *"
+  schedule: "0 40 2 * * *"
   immediate: true
   backupOwnerReference: self
   cluster:

kubernetes/apps/woodpecker/secret.example.yaml

@@ -14,7 +14,7 @@ apiVersion: v1
 kind: Secret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 type: Opaque

kubernetes/apps/woodpecker/server.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server
@@ -84,7 +84,7 @@ spec:
             - name: WOODPECKER_DATABASE_DRIVER
               value: postgres
             - name: WOODPECKER_DATABASE_DATASOURCE
-              value: "host=wp-pg-rw.apps.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
+              value: "host=wp-pg-rw.woodpecker.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
           # AGENT_SECRET + GITEA_CLIENT + GITEA_SECRET come from the SOPS secret.
           envFrom:
             - secretRef:

kubernetes/apps/woodpecker/service.yaml

@@ -2,13 +2,13 @@ apiVersion: v1
 kind: Service
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 spec:
   # Flipped ExternalName -> ClusterIP: ci.renner.uno (via the Ingress, by name)
   # and the in-cluster agent both target this Service now. http:8000 serves the
-  # UI/API; grpc:9000 is how the agent (woodpecker.apps.svc:9000) reaches the
+  # UI/API; grpc:9000 is how the agent (woodpecker.woodpecker.svc:9000) reaches the
   # server. Selects the server pod only (not the agent).
   type: ClusterIP
   selector:

kubernetes/apps/wordpress/deployment.yaml

@@ -0,0 +1,115 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: wordpress
+  namespace: wordpress
+  annotations:
+    secret.reloader.stakater.com/reload: wordpress-secrets
+spec:
+  replicas: 1
+  strategy:
+    # RWO volume, single writer — the rolling default would deadlock on the
+    # new pod waiting for a volume the old pod still holds.
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: wordpress
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: wordpress
+    spec:
+      securityContext:
+        # wp-content must be writable by Apache (www-data, uid/gid 33) for
+        # media uploads and plugin installs.
+        fsGroup: 33
+      containers:
+        - name: wordpress
+          image: wordpress:7.0.4-php8.3-apache
+          ports:
+            - name: http
+              containerPort: 80
+          env:
+            # The operator names the base Service after the MariaDB resource;
+            # the CRD's separate primaryService/secondaryService fields are the
+            # suffixed variants for replicated setups, which this is not
+            # (replicas: 1). Not verified against a running cluster — if it is
+            # wrong the pod fails loudly on connect at startup rather than
+            # degrading quietly. Confirm with `kubectl get svc -n wordpress`
+            # on first deploy.
+            - name: WORDPRESS_DB_HOST
+              value: wordpress-mariadb.wordpress.svc.cluster.local:3306
+            - name: WORDPRESS_DB_NAME
+              value: wordpress
+            - name: WORDPRESS_DB_USER
+              value: wordpress
+            - name: WORDPRESS_DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  # Generated by the operator from the MariaDB CR — never
+                  # stored in Git or OpenBao.
+                  name: wordpress-mariadb-app
+                  key: password
+            - name: WORDPRESS_CONFIG_EXTRA
+              value: |
+                define('WP_HOME', 'https://www.renner.uno');
+                define('WP_SITEURL', 'https://www.renner.uno');
+                define('FORCE_SSL_ADMIN', true);
+                /* Traefik terminates TLS; without this WordPress sees plain
+                   HTTP and builds http:// URLs into every page and redirect. */
+                if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] === 'https') {
+                    $_SERVER['HTTPS'] = 'on';
+                }
+                /* Core lives in the image, not on the volume — the updater
+                   would only fail against a read-only docroot. Plugin and
+                   theme installs are unaffected. */
+                define('WP_AUTO_UPDATE_CORE', false);
+                /* No add_filter() here. WORDPRESS_CONFIG_EXTRA is eval'd inside
+                   wp-config.php, long before wp-settings.php loads the plugin
+                   API, so any add_filter call is a fatal error on every request
+                   — which is exactly what happened on first deploy. xmlrpc is
+                   handled at the ingress instead, by routing /xmlrpc.php
+                   through the Authentik gate. */
+          envFrom:
+            # The eight WordPress salts. Rotating them in OpenBao invalidates
+            # every session — which is the intended lever, not a side effect.
+            - secretRef:
+                name: wordpress-secrets
+          volumeMounts:
+            - name: content
+              mountPath: /var/www/html/wp-content
+          readinessProbe:
+            httpGet:
+              path: /
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /
+              port: http
+            initialDelaySeconds: 60
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 50m
+              memory: 256Mi
+            limits:
+              memory: 768Mi
+      volumes:
+        - name: content
+          persistentVolumeClaim:
+            claimName: wordpress-content
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: wordpress
+  namespace: wordpress
+spec:
+  selector:
+    app.kubernetes.io/name: wordpress
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

kubernetes/apps/wordpress/externalsecret.yaml

@@ -0,0 +1,46 @@
+# The eight WordPress salts (AUTH_KEY, SECURE_AUTH_KEY, LOGGED_IN_KEY,
+# NONCE_KEY, AUTH_SALT, SECURE_AUTH_SALT, LOGGED_IN_SALT, NONCE_SALT).
+# Expected keys in OpenBao at homelab/apps/wordpress, named exactly as the
+# image's env vars: WORDPRESS_AUTH_KEY, WORDPRESS_SECURE_AUTH_KEY, ...
+#
+# Deliberately NOT carried over from the old wp-config.php: the Hetzner host
+# is being decommissioned and its salts have been sitting on a VM whose
+# neighbour site was compromised this month. New salts cost one logout.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: wordpress-secrets
+  namespace: wordpress
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: wordpress-secrets
+  dataFrom:
+    - extract:
+        key: apps/wordpress
+---
+# S3 credentials for the MariaDB Backup.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: wordpress
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
+    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
+    # path for this migration — same approach woodpecker took. FOLLOW-UP: give
+    # wordpress its own seed/wordpress/hetzner-creds path once OpenBao can be
+    # written, and repoint this here (the value is the same, so it's a drop-in
+    # swap).
+    - extract:
+        key: seed/paperless/hetzner-creds

kubernetes/apps/wordpress/gate-service.yaml

@@ -0,0 +1,15 @@
+# A plain Ingress cannot point at a Service in another namespace, and the
+# Authentik outposts all live in `authentik`. Same workaround as
+# zone-configurator and paperless: mirror the outpost Service in as an
+# ExternalName.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ak-outpost-wordpress-gate
+  namespace: wordpress
+spec:
+  type: ExternalName
+  externalName: ak-outpost-wordpress-gate.authentik.svc.cluster.local
+  ports:
+    - port: 9000
+      targetPort: 9000

kubernetes/apps/wordpress/ingress.yaml

@@ -0,0 +1,110 @@
+# Two routers on one host. The site itself must stay readable by anyone --
+# it is a family blog -- while the editing surface sits behind Authentik.
+#
+# Why not one gated Ingress with skip_path_regex, the way paperless-gate
+# carves out /static/: that works when the PUBLIC paths are enumerable. Here
+# it is the reverse -- every post URL is public and only two paths are gated
+# -- which would need a negative lookahead. The outpost is Go, and RE2 has no
+# lookahead. So the split happens at the router instead.
+#
+# Traefik orders routers by rule length, so PathPrefix(`/wp-admin`) is
+# evaluated before PathPrefix(`/`); no explicit priority is needed.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: wordpress
+  namespace: wordpress
+  labels:
+    dns.renner.uno/public: "true"
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: www.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: wordpress
+                port:
+                  number: 80
+  tls:
+    - secretName: wordpress-tls
+      hosts:
+        - www.renner.uno
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: wordpress-admin
+  namespace: wordpress
+  labels:
+    # Same label as the public Ingress on purpose. external-dns-unifi runs
+    # with --label-filter=dns.renner.uno/public!=true, so an unlabelled
+    # Ingress on this host would quietly gain a UDM A-record pointing LAN
+    # clients straight at the Traefik VIP -- a second, unintended path to
+    # the same host that skips the public route entirely.
+    dns.renner.uno/public: "true"
+  # No cert-manager annotation and no tls block: the public Ingress above
+  # already owns the certificate for this host. Declaring it twice would
+  # create two Certificates racing for the same wordpress-tls Secret.
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: www.renner.uno
+      http:
+        paths:
+          # The outpost's own OAuth leg. Without this the gate is a dead end,
+          # not a gate: hitting /wp-admin redirects to
+          # /outpost.goauthentik.io/start on THIS host, which otherwise falls
+          # through to the public `/` router and lands on WordPress's 404
+          # page — verified exactly that way on first deploy. paperless needs
+          # the same route and has it as a separate Ingress on auth.renner.uno
+          # (apps/authentik/paperless-gate-ingress.yaml); here the flow stays
+          # on www.renner.uno, so it belongs in this rule set.
+          #
+          # It is not an authentication hole: these endpoints ARE the login
+          # flow, and the outpost serves them itself.
+          - path: /outpost.goauthentik.io
+            pathType: Prefix
+            backend:
+              service:
+                name: ak-outpost-wordpress-gate
+                port:
+                  number: 9000
+          # Covers admin-ajax.php too. That is only safe because the gift
+          # registry plugin -- the sole thing on this site that ever emitted
+          # an ajaxurl -- is being dropped in the migration. Verified against
+          # the live site: 121 admin-ajax hits in the current log window, not
+          # one of them a 200.
+          - path: /wp-admin
+            pathType: Prefix
+            backend:
+              service:
+                name: ak-outpost-wordpress-gate
+                port:
+                  number: 9000
+          - path: /wp-login.php
+            pathType: Exact
+            backend:
+              service:
+                name: ak-outpost-wordpress-gate
+                port:
+                  number: 9000
+          # xmlrpc.php would otherwise be served by the public `/` prefix
+          # router — an earlier version of this file claimed it was "blocked
+          # at the ingress" while nothing blocked it at all. There is no deny
+          # verb in an Ingress, so it goes through the gate instead: anonymous
+          # callers get Authentik, not the XML-RPC endpoint. Nothing on this
+          # site uses it (no Jetpack, no mobile app, no pingbacks worth
+          # keeping), so no legitimate caller is affected.
+          - path: /xmlrpc.php
+            pathType: Exact
+            backend:
+              service:
+                name: ak-outpost-wordpress-gate
+                port:
+                  number: 9000

kubernetes/apps/wordpress/kustomization.yaml

@@ -0,0 +1,9 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - externalsecret.yaml
+  - mariadb.yaml
+  - pvc.yaml
+  - deployment.yaml
+  - gate-service.yaml
+  - ingress.yaml

kubernetes/apps/wordpress/mariadb.yaml

@@ -0,0 +1,86 @@
+# The database for www.renner.uno. Named `wordpress-mariadb`, not `wp-*`:
+# `wp-pg` in the woodpecker namespace is Woodpecker's Postgres, and reusing
+# the prefix for an unrelated engine invites exactly the wrong guess.
+#
+# Mirrors the CNPG convention: the operator lives in `database`, the database
+# resource lives with its consumer (ak-pg in authentik, im-pg in immich).
+apiVersion: k8s.mariadb.com/v1alpha1
+kind: MariaDB
+metadata:
+  name: wordpress-mariadb
+  namespace: wordpress
+  annotations:
+    # Data guard — a Kustomization rebuild must never take the database with it.
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  # generate: true lets the operator create both Secrets on first reconcile.
+  # Neither password is ever typed by a human or parked in OpenBao, so there
+  # is nothing to rotate by hand and nothing to leak through Git.
+  rootPasswordSecretKeyRef:
+    name: wordpress-mariadb-root
+    key: password
+    generate: true
+  username: wordpress
+  passwordSecretKeyRef:
+    name: wordpress-mariadb-app
+    key: password
+    generate: true
+  database: wordpress
+  port: 3306
+  replicas: 1
+
+  storage:
+    size: 5Gi
+    storageClassName: ceph-block
+  # The source database is 527 KB. 5Gi is not a sizing estimate, it is the
+  # smallest allocation that leaves room for binlogs and a decade of posts.
+
+  myCnf: |
+    [mariadb]
+    bind-address=*
+    default_storage_engine=InnoDB
+    innodb_buffer_pool_size=256M
+    # WordPress writes utf8mb4 throughout; the source dump carries
+    # utf8mb4_unicode_520_ci on every table that survives the migration.
+    character_set_server=utf8mb4
+    collation_server=utf8mb4_unicode_520_ci
+    max_allowed_packet=64M
+
+  resources:
+    requests:
+      cpu: 100m
+      memory: 512Mi
+    limits:
+      memory: 1Gi
+---
+# Logical backup to Hetzner Object Storage. Velero only captures PVC data,
+# which for a running database is crash-consistent, not transaction-consistent
+# — that is a snapshot, not a backup you would want to restore from.
+apiVersion: k8s.mariadb.com/v1alpha1
+kind: Backup
+metadata:
+  name: wordpress-mariadb-backup
+  namespace: wordpress
+spec:
+  mariaDbRef:
+    name: wordpress-mariadb
+  schedule:
+    # 02:45, ahead of the Velero window that starts at 03:15.
+    cron: "45 2 * * *"
+    suspend: false
+  maxRetention: 720h # 30d, matching the CNPG ObjectStore retentionPolicy
+  compression: gzip
+  storage:
+    s3:
+      bucket: renner-uno-k8s-mariadb
+      prefix: wordpress
+      endpoint: fsn1.your-objectstorage.com
+      region: fsn1
+      accessKeyIdSecretKeyRef:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKeySecretKeyRef:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      tls:
+        enabled: true

kubernetes/apps/wordpress/pvc.yaml

@@ -0,0 +1,35 @@
+# Only wp-content is persisted, not the whole docroot. WordPress core comes
+# from the pinned image, so upgrading the site means bumping a tag in Git
+# rather than clicking "update" in wp-admin — which is the entire point of
+# moving off the Hetzner VM, where core was a mutable pile of PHP on disk.
+#
+# Consequence to be aware of: WordPress's own core updater will fail against
+# a read-only core. Plugin and theme installs still work, because those land
+# in wp-content.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: wordpress-content
+  namespace: wordpress
+  labels:
+    # The Velero Schedule selects ON this label rather than excluding the
+    # database PVC by one the mariadb-operator is merely assumed to set. If
+    # that assumption were wrong the exclusion would fail silently and Velero
+    # would snapshot the live database volume; selecting a label this repo
+    # owns cannot fail that way. Same direction as openbao-daily, which picks
+    # its snapshot PVC by app.kubernetes.io/component.
+    app.kubernetes.io/name: wordpress
+    app.kubernetes.io/component: content
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteOnce
+  # RWO is sufficient: a single replica with strategy Recreate. ceph-block is
+  # the cluster default; named explicitly so a default change can't silently
+  # move the media library onto another class.
+  storageClassName: ceph-block
+  resources:
+    requests:
+      # The migrated wp-content is 29 MB compressed, 13 MB of it uploads.
+      storage: 10Gi

kubernetes/clusters/homelab/infrastructure.yaml

@@ -48,6 +48,10 @@ spec:
       kind: HelmRelease
       name: cloudnative-pg # DB operator — seed jobs create databases
       namespace: database
+    - apiVersion: helm.toolkit.fluxcd.io/v2
+      kind: HelmRelease
+      name: mariadb-operator # the MySQL half — WordPress can't use CNPG
+      namespace: database
   # decryption block dropped 2026-07-13: no *.sops.yaml left under this
   # path — secrets come from OpenBao via ESO (MIGRATION-sops-openbao.md).
 ---