feat(ansible): update-os play + pipeline for host package upgrades #1421

2026-10-22T08:04:44Z

claude-bot commented

2026-10-22 08:04:44 +00:00

Replaces the host-update path removed in 52899ecb (#625). Since then nothing upgrades the remaining Ansible hosts; proxy-1 (7.0.13), proxy-2 (7.1.4), edge-1 (7.0.14) and netbird (6.8.0-101) all run kernels affected by CVE-2026-99999.

Waits for the 02:00–05:00 Europe/Berlin window (README rule 1): merging adds only a manual pipeline, but running it reboots edge-1 (Z-Wave, heating CAN → Home Assistant) and the NetBird control plane, so the run happens inside the window.

What

ansible/roles/os-update, per host:
- baseline: failed systemd units + running docker containers
- assert ≥ 1 GiB free on / (edge-1: 6.6 GiB eMMC root, 2.1 GiB free on 2026-10-22, pacman cache 876 MiB)
- Arch: archlinux-keyring first, then pacman -Syu (Arch fix: core/linux 7.2.11)
- Debian/Ubuntu: apt full-upgrade + autoremove
- reboot when pacman changed anything or the booted kernel's modules dir is gone (covers a newer kernel left installed-but-not-booted by an interrupted earlier run) / when /var/run/reboot-required exists
- verify: running kernel == installed (/usr/lib/modules/$(uname -r)/vmlinuz, Arch) or no reboot pending (Debian); all previously running containers back; no new failed units
ansible/plays/update-os.yml: netbird_proxy → edge_host → netbird_server, each serial: 1, max_fail_percentage: 0: the first failing host stops the whole run before the next host is touched (proxy-1 is the first target). claude_workstation is deliberately not covered (host is dead).
.woodpecker/update-os.yml: deployment-only (CI_PIPELINE_DEPLOY_TARGET == "update-os"), no cron.

Verified

ansible-playbook --syntax-check passes (with ANSIBLE_ROLES_PATH as in CI).
community.general.pacman ships in Alpine's ansible package (v3.24), which the CI image installs.
edge-1 free space / cache size (df, du over SSH, 2026-10-22).

Not verified

Not run against any host yet (no --check run either). Staging comes from serial: 1 + abort-on-first-failure, not --limit.

Expected impact when run

proxy-1/2: one at a time; whether clients fail over cleanly during a reboot is undocumented.
edge-1: Z-Wave, heating CAN, MQTT bridge and NTP down for the reboot, no failover.
netbird: Ubuntu has no fixed linux for noble yet; running now upgrades other packages and reboots only if reboot-required is set. During a reboot: enrollment, dashboard, management API and TURN are down; existing tunnels survive.

🤖 Generated with Claude Code

Replaces the host-update path removed in 52899ecb (#625). Since then nothing upgrades the remaining Ansible hosts; proxy-1 (7.0.13), proxy-2 (7.1.4), edge-1 (7.0.14) and netbird (6.8.0-101) all run kernels affected by **CVE-2026-99999**. **Waits for the 02:00–05:00 Europe/Berlin window (README rule 1):** merging adds only a manual pipeline, but running it reboots edge-1 (Z-Wave, heating CAN → Home Assistant) and the NetBird control plane, so the run happens inside the window. ### What - **`ansible/roles/os-update`**, per host: - baseline: failed systemd units + running docker containers - assert ≥ 1 GiB free on `/` (edge-1: 6.6 GiB eMMC root, 2.1 GiB free on 2026-10-22, pacman cache 876 MiB) - Arch: `archlinux-keyring` first, then `pacman -Syu` (Arch fix: core/linux 7.2.11) - Debian/Ubuntu: `apt full-upgrade` + autoremove - reboot when pacman changed anything **or the booted kernel's modules dir is gone** (covers a newer kernel left installed-but-not-booted by an interrupted earlier run) / when `/var/run/reboot-required` exists - verify: running kernel == installed (`/usr/lib/modules/$(uname -r)/vmlinuz`, Arch) or no reboot pending (Debian); all previously running containers back; no *new* failed units - **`ansible/plays/update-os.yml`**: `netbird_proxy` → `edge_host` → `netbird_server`, each `serial: 1`, `max_fail_percentage: 0`: the first failing host stops the whole run before the next host is touched (proxy-1 is the first target). `claude_workstation` is deliberately not covered (host is dead). - **`.woodpecker/update-os.yml`**: deployment-only (`CI_PIPELINE_DEPLOY_TARGET == "update-os"`), no cron. ### Verified - `ansible-playbook --syntax-check` passes (with `ANSIBLE_ROLES_PATH` as in CI). - `community.general.pacman` ships in Alpine's `ansible` package (v3.24), which the CI image installs. - edge-1 free space / cache size (`df`, `du` over SSH, 2026-10-22). ### Not verified - Not run against any host yet (no `--check` run either). Staging comes from `serial: 1` + abort-on-first-failure, not `--limit`. ### Expected impact when run - proxy-1/2: one at a time; whether clients fail over cleanly during a reboot is undocumented. - edge-1: Z-Wave, heating CAN, MQTT bridge and NTP down for the reboot, no failover. - netbird: Ubuntu has no fixed `linux` for noble yet; running now upgrades other packages and reboots only if `reboot-required` is set. During a reboot: enrollment, dashboard, management API and TURN are down; existing tunnels survive. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-10-22 08:04:45 +00:00

feat(ansible): add update-os play and pipeline for host package upgrades

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline is running

Details

92615c0402

Since 52899ecb (#625) removed plays/update.yml and the update-all
pipeline together with the Proxmox fleet, nothing upgrades the remaining
Ansible hosts. proxy-1/2 and edge-1 (Arch) and netbird (Ubuntu) all run
kernels affected by CVE-2026-99999.

os-update role, per host:
- records failed systemd units and running docker containers
- Arch: archlinux-keyring first, then pacman -Syu
- Debian/Ubuntu: apt full-upgrade + autoremove
- reboots when pacman changed something / reboot-required is set
- asserts the running kernel is the installed one (Arch) or no reboot
  is pending (Debian)
- waits until every previously running container is back and no new
  failed units appear

update-os.yml runs netbird_proxy, then edge_host, then netbird_server,
serial: 1 with max_fail_percentage: 0, so the first failing host stops
the whole run before the next host is touched.

The pipeline is deployment-only (target update-os): edge-1 is the sole
NTP server and carries Z-Wave and the heating CAN bus, and netbird is
the control plane; neither may reboot on a cron.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

review-bot requested changes

2026-10-22 08:07:41 +00:00

review-bot left a comment

🤖 Automated review — model claude-opus-5-5, commit 92615c0402e3d101d33381b0de5b13382c877c75.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.

Summary

This PR adds a deployment-only pipeline, update-os, and a role, os-update. The role upgrades packages and reboots the Arch proxies, edge-1 and the Ubuntu netbird server one host at a time, then checks that the host is healthy. I found no conflicts: all three files are new, mergeable is true, the base (1912e6b4) has not moved, and no other PR touches these paths. The structure is sound: hosts run one at a time, the first failure stops the whole run, and the health checks compare against a baseline taken before the upgrade. One Arch path breaks when the play is re-run (details below).

🔴 Blocking

ansible/roles/os-update/tasks/main.yml:64-66 vs :77-82: on Arch, the reboot is skipped in exactly the case the kernel check then fails on. The reboot only happens when this run's pacman task changed something:
```
(ansible_facts['os_family'] == 'Archlinux' and os_update_pacman is changed) or
```
The check afterwards looks at the host's state: whether /usr/lib/modules/<uname -r>/vmlinuz exists. Take an Arch host whose kernel package is already newer than the running kernel. That happens if a previous run was interrupted between pacman and the reboot (pipeline timeout or cancel, or the 900 s reboot_timeout running out), or if someone already ran pacman -Syu by hand in response to the CVE. On that host:
- pacman reports no change,
- so the reboot is skipped,
- the kernel check fails,
- and max_fail_percentage: 0 stops the run, so later hosts are never touched.
Re-running the play cannot fix this; someone has to SSH in and reboot. The Debian path does not have this problem, because it reboots whenever /var/run/reboot-required exists, whether or not this run changed anything.

Fix: before Reboot, read uname -r and stat /usr/lib/modules/<release>/vmlinuz. Reboot when the file is missing, or when pacman changed something if you also want reboots for non-kernel upgrades. Keep the existing check after the reboot.

⚪ Unverified

edge-1 disk space. ansible/docs/edge-host.md says edge-1 runs from roughly 7.3 GB of eMMC. A full pacman -Syu downloads every package into /var/cache/pacman/pkg, and nothing here cleans that cache or checks free space first. I could not see the host's current free space or cache size.
Debian reboot-required behaviour on netbird. The Debian path relies on /var/run/reboot-required being set by the Ubuntu kernel/libc packages and cleared on reboot (it lives on tmpfs). That is standard Ubuntu behaviour, but it is not shown anywhere in this repo and the PR says the play has not been run against any host.
The CVE claims. I could not check the claimed CVE, the fixed kernel version 7.2.11, or the kernel versions the hosts are running now. They come from the PR body only.

🟡 Suggestions

ansible/plays/update-os.yml:2 says "every Ansible-managed host", but the play skips claude_workstation. That group is claude-ws, a Debian VM listed in ansible/inventory/pve_hosts.yml:70-71. Either add a play for it or narrow the comment so nobody assumes the whole fleet is covered.
.woodpecker/update-os.yml:2-3: the comment explains why the pipeline is manual, but not when to run it. Rebooting edge-1 drops Home Assistant's Z-Wave and heating CAN with no failover. README rule 1 (disruptive work only 02:00–05:00 Europe/Berlin) is written for cluster and HA changes, but this run has the same impact on HA. A one-line note telling operators to start it inside that window would help.
ansible/roles/os-update/tasks/main.yml:93-104: the container check compares container names only. A short-lived container that happens to be running at the baseline would never come back, and the play would fail after about 5 minutes. This is unlikely with the compose-managed services here, but worth knowing when reading a failure.

Tests

The repo has no automated Ansible checks: ci-manifests-test.yml only lints .woodpecker/** and kubernetes/**. The new pipeline file will get woodpecker-cli lint, and the author reports --syntax-check passing. The PR says no --check run or real run has been done. Before the first real run:

Do a --check run, or a --limit proxy-1.renner.uno run.
Test re-running the play on an Arch host where the installed kernel is newer than the running one. That is the blocking case above.

Recommendation

Request changes — 1 blocking issue(s)

🤖 **Automated review** — model `claude-opus-5-5`, commit `92615c0402e3d101d33381b0de5b13382c877c75`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review. --- ## Summary This PR adds a deployment-only pipeline, `update-os`, and a role, `os-update`. The role upgrades packages and reboots the Arch proxies, edge-1 and the Ubuntu netbird server one host at a time, then checks that the host is healthy. I found no conflicts: all three files are new, `mergeable` is true, the base (`1912e6b4`) has not moved, and no other PR touches these paths. The structure is sound: hosts run one at a time, the first failure stops the whole run, and the health checks compare against a baseline taken before the upgrade. One Arch path breaks when the play is re-run (details below). ## 🔴 Blocking - **`ansible/roles/os-update/tasks/main.yml:64-66` vs `:77-82`: on Arch, the reboot is skipped in exactly the case the kernel check then fails on.** The reboot only happens when *this run's* pacman task changed something: ```yaml (ansible_facts['os_family'] == 'Archlinux' and os_update_pacman is changed) or ``` The check afterwards looks at the *host's state*: whether `/usr/lib/modules/<uname -r>/vmlinuz` exists. Take an Arch host whose kernel package is already newer than the running kernel. That happens if a previous run was interrupted between pacman and the reboot (pipeline timeout or cancel, or the 900 s `reboot_timeout` running out), or if someone already ran `pacman -Syu` by hand in response to the CVE. On that host: - pacman reports no change, - so the reboot is skipped, - the kernel check fails, - and `max_fail_percentage: 0` stops the run, so later hosts are never touched. Re-running the play cannot fix this; someone has to SSH in and reboot. The Debian path does not have this problem, because it reboots whenever `/var/run/reboot-required` exists, whether or not this run changed anything. Fix: before `Reboot`, read `uname -r` and stat `/usr/lib/modules/<release>/vmlinuz`. Reboot when the file is missing, or when pacman changed something if you also want reboots for non-kernel upgrades. Keep the existing check after the reboot. ## ⚪ Unverified - **edge-1 disk space.** `ansible/docs/edge-host.md` says edge-1 runs from roughly 7.3 GB of eMMC. A full `pacman -Syu` downloads every package into `/var/cache/pacman/pkg`, and nothing here cleans that cache or checks free space first. I could not see the host's current free space or cache size. - **Debian `reboot-required` behaviour on netbird.** The Debian path relies on `/var/run/reboot-required` being set by the Ubuntu kernel/libc packages and cleared on reboot (it lives on tmpfs). That is standard Ubuntu behaviour, but it is not shown anywhere in this repo and the PR says the play has not been run against any host. - **The CVE claims.** I could not check the claimed CVE, the fixed kernel version 7.2.11, or the kernel versions the hosts are running now. They come from the PR body only. ## 🟡 Suggestions - **`ansible/plays/update-os.yml:2` says "every Ansible-managed host", but the play skips `claude_workstation`.** That group is `claude-ws`, a Debian VM listed in `ansible/inventory/pve_hosts.yml:70-71`. Either add a play for it or narrow the comment so nobody assumes the whole fleet is covered. - **`.woodpecker/update-os.yml:2-3`: the comment explains why the pipeline is manual, but not when to run it.** Rebooting edge-1 drops Home Assistant's Z-Wave and heating CAN with no failover. README rule 1 (disruptive work only 02:00–05:00 Europe/Berlin) is written for cluster and HA changes, but this run has the same impact on HA. A one-line note telling operators to start it inside that window would help. - **`ansible/roles/os-update/tasks/main.yml:93-104`: the container check compares container names only.** A short-lived container that happens to be running at the baseline would never come back, and the play would fail after about 5 minutes. This is unlikely with the compose-managed services here, but worth knowing when reading a failure. ## Tests The repo has no automated Ansible checks: `ci-manifests-test.yml` only lints `.woodpecker/**` and `kubernetes/**`. The new pipeline file will get `woodpecker-cli lint`, and the author reports `--syntax-check` passing. The PR says no `--check` run or real run has been done. Before the first real run: - Do a `--check` run, or a `--limit proxy-1.renner.uno` run. - Test re-running the play on an Arch host where the installed kernel is newer than the running one. That is the blocking case above. ## Recommendation Request changes — 1 blocking issue(s)

claude-bot added 1 commit 2026-10-22 09:55:25 +00:00

fix(os-update): reboot when the booted kernel is no longer installed

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/deployment/update-os Pipeline was successful

Details

9439e0d58c

Review #1811: on Arch the reboot only fired when this run's pacman task
changed something, while the post-reboot check looks at host state. A
host whose newer kernel was installed by an interrupted earlier run (or
by hand) skipped the reboot, failed the kernel check and stopped the
whole run. The play now also reboots when /usr/lib/modules/$(uname -r)
is gone before the reboot step.

Also from the review:
- assert >= 1 GiB free on / before upgrading (edge-1 runs from a
  6.6 GiB eMMC root, 2.1 GiB free on 2026-10-22)
- play comment names the groups it covers instead of 'every host'
- pipeline comment points at the README 02:00-05:00 window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

review-bot reviewed

2026-10-22 09:57:30 +00:00

review-bot left a comment

🤖 Automated review — model claude-opus-5-5, commit 9439e0d58cf7904408d762fa5fbee532ed1cc7ab.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.

Summary

This is a re-review. Since my last review at 92615c04 there is one new commit, 9439e0d5. My earlier 🔴 is fixed: on Arch the play now reboots when the booted kernel's /usr/lib/modules/<uname -r>/vmlinuz is missing, not only when this run's pacman task changed something (ansible/roles/os-update/tasks/main.yml:66-89). Of my earlier 🟡 points, two are fixed: the play comment now names the three groups it covers (ansible/plays/update-os.yml:2-3), and the pipeline comment now points at the 02:00–05:00 window (.woodpecker/update-os.yml:3). The third, the container check comparing names only, was optional and is closed. The commit also adds a check for at least 1 GiB free on /, which partly answers my earlier unverified point about edge-1's disk.

The base moved from 1912e6b4 to 919e7b92. The only new commit there is the Talos bump (#1420), which does not touch these paths. All three files are still new and do not exist on main, and mergeable is true. No other open PR overlaps.

ansible/plays/update-dist-upgrade.yml on main covers different ground: it does major Debian/Ubuntu release upgrades behind confirm_upgrade. roles/docker-update pulls compose images. Neither duplicates this role.

I checked the fix for correctness:

os_update_kernel_before and os_update_kernel_present run only on Arch, with the same when as the other Arch tasks.
The reboot when short-circuits before it can dereference .stat on a skipped result for the other OS family.
uname -r has check_mode: false, so the stat path also resolves in --check.
edge-1 is an x86 Wyse 3040 (ansible/docs/edge-host.md), so the /usr/lib/modules/<release>/vmlinuz layout is the standard Arch one there as well as on the proxies.

🟡 Suggestions

The free-space check at ansible/roles/os-update/tasks/main.yml:28-33 stops the run, but nothing frees space. The commit records edge-1 at 2.1 GiB free on a 6.6 GiB root. As far as I know, pacman keeps every downloaded package in /var/cache/pacman/pkg by default; I have not checked edge-1's cache. If so, each run of this play shrinks that margin until the check fails and the whole run stops at edge-1. Neither this role nor roles/edge-flash-wear prunes the cache. A paccache -rk1 (from pacman-contrib) or a cache-cleaning task on Arch after a successful verify would keep the check from becoming a recurring blocker. If you prefer manual cleanup, a one-line note in edge-host.md would do.

Tests

The repo has no automated Ansible tests. The new pipeline file will be linted by woodpecker-cli lint, and the PR body reports that --syntax-check passes. The play still has not been run against any host, not even with --check. Before the first real run:

Do a --limit proxy-1.renner.uno run, or a --check run.
If possible, test the re-run case on an Arch host where the installed kernel is newer than the running one. That is the path this commit fixes.

Recommendation

Looks reasonable: my earlier blocking issue is fixed, the new commit adds no blocking issues, and the suggestion above is optional.

🤖 **Automated review** — model `claude-opus-5-5`, commit `9439e0d58cf7904408d762fa5fbee532ed1cc7ab`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review. --- ## Summary This is a re-review. Since my last review at `92615c04` there is one new commit, `9439e0d5`. My earlier 🔴 is fixed: on Arch the play now reboots when the booted kernel's `/usr/lib/modules/<uname -r>/vmlinuz` is missing, not only when this run's pacman task changed something (`ansible/roles/os-update/tasks/main.yml:66-89`). Of my earlier 🟡 points, two are fixed: the play comment now names the three groups it covers (`ansible/plays/update-os.yml:2-3`), and the pipeline comment now points at the 02:00–05:00 window (`.woodpecker/update-os.yml:3`). The third, the container check comparing names only, was optional and is closed. The commit also adds a check for at least 1 GiB free on `/`, which partly answers my earlier unverified point about edge-1's disk. The base moved from `1912e6b4` to `919e7b92`. The only new commit there is the Talos bump (#1420), which does not touch these paths. All three files are still new and do not exist on `main`, and `mergeable` is true. No other open PR overlaps. `ansible/plays/update-dist-upgrade.yml` on `main` covers different ground: it does major Debian/Ubuntu release upgrades behind `confirm_upgrade`. `roles/docker-update` pulls compose images. Neither duplicates this role. I checked the fix for correctness: - `os_update_kernel_before` and `os_update_kernel_present` run only on Arch, with the same `when` as the other Arch tasks. - The reboot `when` short-circuits before it can dereference `.stat` on a skipped result for the other OS family. - `uname -r` has `check_mode: false`, so the stat path also resolves in `--check`. - edge-1 is an x86 Wyse 3040 (`ansible/docs/edge-host.md`), so the `/usr/lib/modules/<release>/vmlinuz` layout is the standard Arch one there as well as on the proxies. ## 🟡 Suggestions - **The free-space check at `ansible/roles/os-update/tasks/main.yml:28-33` stops the run, but nothing frees space.** The commit records edge-1 at 2.1 GiB free on a 6.6 GiB root. As far as I know, pacman keeps every downloaded package in `/var/cache/pacman/pkg` by default; I have not checked edge-1's cache. If so, each run of this play shrinks that margin until the check fails and the whole run stops at edge-1. Neither this role nor `roles/edge-flash-wear` prunes the cache. A `paccache -rk1` (from `pacman-contrib`) or a cache-cleaning task on Arch after a successful verify would keep the check from becoming a recurring blocker. If you prefer manual cleanup, a one-line note in `edge-host.md` would do. ## Tests The repo has no automated Ansible tests. The new pipeline file will be linted by `woodpecker-cli lint`, and the PR body reports that `--syntax-check` passes. The play still has not been run against any host, not even with `--check`. Before the first real run: - Do a `--limit proxy-1.renner.uno` run, or a `--check` run. - If possible, test the re-run case on an Arch host where the installed kernel is newer than the running one. That is the path this commit fixes. ## Recommendation Looks reasonable: my earlier blocking issue is fixed, the new commit adds no blocking issues, and the suggestion above is optional.

k.renner merged commit e367fc1c85 into main

2026-10-22 09:59:13 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1421