Compare commits
14
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ebd9625926 | ||
|
|
6716f7dc50 | ||
|
|
d505503616 | ||
|
|
2ba950497d | ||
|
|
664bc72eb4 | ||
|
|
14e45a540c | ||
|
|
4965f53990 | ||
|
|
e7c70ea368 | ||
|
|
4187aa91cb | ||
|
|
6ff5cb7d7f | ||
|
|
c8b0d1cd4b | ||
|
|
19462a3792 | ||
|
|
75710eab2e | ||
|
|
b76a948e28 |
@@ -170,8 +170,8 @@ role keeps logs off local flash and monitors wear:
|
||||
WAL/positions sit on `/run/alloy` (tmpfs), so the agent writes nothing to
|
||||
the eMMC either. Net effect: no log writes hit local flash — the trade-off
|
||||
is that logs are RAM-only and lost on reboot unless already shipped.
|
||||
- **Loki endpoint:** the unauthenticated srv-network Loki
|
||||
(`loki.srv.renner.uno:3100`) is firewalled off the IoT VLAN, so Alloy pushes
|
||||
- **Loki endpoint:** the old srv-network Loki
|
||||
(`loki.srv.renner.uno:3100`) no longer exists — Loki moved into the cluster, so Alloy pushes
|
||||
to the authenticated cluster ingress `https://loki.renner.uno/loki/api/v1/push`
|
||||
with basic-auth (`edge_loki_username` / `vault_edge_loki_password`).
|
||||
**Delivery only works once the `edge-1` htpasswd line has reconciled into
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
$ANSIBLE_VAULT;1.1;AES256
|
||||
32373134633930343261336362373335623735656634313034393731393661643361346634623439
|
||||
3561363963333533313264643036663731656465343162330a333964383462373934303766323533
|
||||
63613065313063626261383236396363656565623538666565646661353766643133306334396561
|
||||
3961376234623236620a616339326238366538333666316538396465666365373834313064653963
|
||||
38623233616633383435373836613438333763343932636239386263306238376137643531373161
|
||||
35386437313465643437343832623935353238316565366537646530323233363837656631383962
|
||||
32653332333162623966366163633732356637323461613966313263313066643966373335313162
|
||||
30306362346561633566316166313164323635356363643034663933363464663562343136646137
|
||||
66643536363734316533663032643637356136363237376365613330633961616665323430633135
|
||||
36323733633464306330666232663433616533373466633939353061663136666665323432373666
|
||||
36393930333435333238306238373931393232393861333334643139306138653336633534626362
|
||||
61323038303838396438
|
||||
@@ -1,42 +1,29 @@
|
||||
$ANSIBLE_VAULT;1.1;AES256
|
||||
39333932383736353162346462663539633763663565366639646463316636333364386365363863
|
||||
3034356438663432363366633537373738363061633764300a393236373834323237623434353361
|
||||
32346331666234616335663432386163316161616230396265373337613035323863393663333665
|
||||
6463623335626238640a306666616366363130363534623032353730313865343537373931343433
|
||||
61636634306565356566303431623533623734343466343636376664646539363363383733666638
|
||||
39316362393066666435326438323036353735666630306532353433663163303231376565313838
|
||||
33616137303261366330653862613934303138653634323063363232303636346336303337333537
|
||||
33643030613662343765303638303339336564633063626462656265646536616563303565323266
|
||||
36373765663732666436363866623133366337646265616333633536666630396465363438353865
|
||||
37383130373532303131646530353236383637613034616433383333333234353137633166663866
|
||||
30366265323537623863336461313666656662383535623435306231363763383364306335623938
|
||||
34376363633762303664356262383837313866633362663037616335626135376366613336356132
|
||||
39323166366437386236386237396236356631623133373866663831656330313936363361656465
|
||||
31373663653736653438643165383338333034613139303333373937626666376432613665656564
|
||||
61616365663033633438666562363530323862613436663964356339373635356366386264373437
|
||||
36646138313838666561646331316265383637313237616632643831303835313663363034633862
|
||||
62623064383539366366386634313938323833323163316365356430633531306134366630623261
|
||||
35656461306566666531366364323834663737313835666264316132376638333436343136656233
|
||||
39343338653264636435383736613763633531356438643464633164633766383435323331633630
|
||||
63316436343863383561653631333434613761343863643465366139353262333463646263633566
|
||||
38306266393439346265666133343566353164353765366331343631666163613262363135643938
|
||||
31643538363136376462633431643135373932373932303435303365383063646236656532643263
|
||||
34663637616535373235336537633339303263383035323264303665643562336431303436653435
|
||||
37613162353133636361623431356361646433313161316163333766336531363164316334343061
|
||||
38303263643263386662383731656537393731356663353861316630353464363932346666323037
|
||||
31343933346530306461363665383235303239373234336631316531353862643937393235643134
|
||||
32303265306331336665653162383639353634373462666461643463633531373366633438353538
|
||||
31376437386538613531323231396437363434663236663635666638373137353434373733633234
|
||||
66363732393166633534316439343561383234636661316439366362326631656134346264623432
|
||||
35626332646162386236336434393533626435313462303635313439326163636330636133613335
|
||||
38636664376166613461316466323161386436396235376534393063663532333538336164643730
|
||||
30636137633663666532666466363730633137396363653832643837366266626461376636333736
|
||||
33343832326566353264353266376664643061306630633038313534386563616236636438393434
|
||||
61643030616531363963313337646263613366663462326561393261383732376231383036303633
|
||||
66633164623232343537316163636666653932646266643561303561323536313532666538643031
|
||||
61653663613139383234656465393637323031666462663165323138303164313461623832323236
|
||||
32373866636432653135623330333464626661396465353133373965376436353138313866633138
|
||||
30376339626165386262396139393737343036333437623365313534636239373830663437643531
|
||||
63643233663137646335343338663661623431636162306363363839306631356638356462643336
|
||||
62643861336131616433666265316538346264376236386434343539343839313130633535616236
|
||||
32336632306165376635313130346436663939303432353966323662326335666564
|
||||
30353730633631376539373334376531353263396339313464373538396231616166653738383161
|
||||
6438656132616365646463653464396236313437323537620a363632643061333036353537626461
|
||||
34393065613836343536383565303066326133363937386263653961663666643430663139653961
|
||||
6462363235343063350a636165376232366636363233376637636231363839343738346134623562
|
||||
63663839326435303964313730336436396365343163343366613966323938656263303166326435
|
||||
65303931393233623064343763633161353161643635396365316362333137353732346366643162
|
||||
36386132386638623333623839373161386330353635346331393735366662393935353264646439
|
||||
36386539393432373765396562613033656339646532323864366230393638313161383363636636
|
||||
32353266376563393239393734316461313566386233333164623135303132346164353165323338
|
||||
62306333363838393236346435303339393639623231353232643966613163313134336265633636
|
||||
64366539326639303362393933303465376532626663346431613563353032383037363532646637
|
||||
38376266323537343437643635343634636164393230313632663533666434306134366261383230
|
||||
66383135303565383763646331383338646339613539623835366438393635363832366134316531
|
||||
63636466653239383231616139663231623636376564376234383762383066613030313937616533
|
||||
33396135616236643034303236353230376434313365326633303961303232363766633161353434
|
||||
64333832623535343733353462313862343934633363616237353331386264386632643538333939
|
||||
34636437393934326131363236346636663665613537343134313636623439356665636663613938
|
||||
37346634356164343537396363663730306232623964636635333333653738383538343236383535
|
||||
66326462373964333234343435336630643930333433393034303439366631386666393330343238
|
||||
61336237373236313533333732343064336637326636663433396362346631343165656163313561
|
||||
30383331326231383539343639343130323731333432306630353835343338613861376631646264
|
||||
66383539616663376237663234313431313064663566313566366664386439316161353135306338
|
||||
35373962636537653535333935306564303235356232313463386237353737313736386463396164
|
||||
62666165336539316565623634326562356434313833643833353635303361353937666663356263
|
||||
35653665323736386263333437666432623332333936373938376637633463653930326466383139
|
||||
38336239646135316565356564373731386662383234386433633632393165343331353835623437
|
||||
30663138633430356334353832633434633939373337663437326263336537356633313763353337
|
||||
64373764666430643433
|
||||
|
||||
@@ -11,12 +11,6 @@
|
||||
ignore_unreachable: true
|
||||
gather_facts: true
|
||||
|
||||
vars:
|
||||
# Monitoring agent endpoints
|
||||
loki_endpoint: "http://loki.srv.renner.uno:3100"
|
||||
prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
|
||||
|
||||
roles:
|
||||
- common
|
||||
- qemu-guest-agent
|
||||
- monitoring-agent
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
- name: Deploy the Claude Code workstation
|
||||
hosts: claude_workstation
|
||||
become: true
|
||||
# qemu-guest-agent + monitoring-agent (Alloy -> Loki/Prometheus) are applied
|
||||
# qemu-guest-agent is applied
|
||||
# to every inventory host by the umbrella plays/common.yml, same as for the
|
||||
# netbird-proxy host — run that against this host too (see RUNBOOK Step 5).
|
||||
roles:
|
||||
|
||||
@@ -18,9 +18,10 @@
|
||||
#
|
||||
# Self-contained: `certificates` and `common` are the parts of plays/common.yml
|
||||
# that apply to an edge host, carried here so bootstrapping one is a single
|
||||
# unrestricted run over the group. Not carried over: qemu-guest-agent (bare
|
||||
# metal, not VMs) and monitoring-agent (Debian-oriented; these hosts ship logs
|
||||
# via edge-log-shipper).
|
||||
# unrestricted run over the group. Not carried over: qemu-guest-agent, since
|
||||
# these are bare metal rather than VMs. Log shipping is edge-log-shipper's job
|
||||
# and lives below — it pushes to the authenticated cluster ingress with a
|
||||
# per-host credential, which is the only Loki these hosts can reach.
|
||||
#
|
||||
# Order is dependency order: edge-flash-wear before docker, so the journal is
|
||||
# volatile before containers start logging into it.
|
||||
|
||||
@@ -1,38 +0,0 @@
|
||||
---
|
||||
# Deploy monitoring agent to NetBird server (external datacenter)
|
||||
# Uses NetBird mesh IPs to reach Loki and Prometheus
|
||||
# Usage: ansible-playbook -i inventory/ plays/setup-netbird-monitoring.yml
|
||||
|
||||
- name: Deploy Monitoring Agent to NetBird Server
|
||||
hosts: netbird_server
|
||||
become: yes
|
||||
gather_facts: yes
|
||||
|
||||
vars:
|
||||
# Use ARM64 binary for Hetzner ARM server
|
||||
alloy_arch: "linux-arm64"
|
||||
|
||||
# Monitoring endpoints via NetBird mesh
|
||||
loki_endpoint: "http://loki.nb.renner.uno:3100"
|
||||
prometheus_remote_write_endpoint: "http://prometheus.nb.renner.uno:9090/api/v1/write"
|
||||
|
||||
# NetBird server labels
|
||||
alloy_extra_labels:
|
||||
environment: "production"
|
||||
datacenter: "hetzner"
|
||||
type: "netbird-server"
|
||||
service: "netbird"
|
||||
|
||||
roles:
|
||||
- monitoring-agent
|
||||
|
||||
post_tasks:
|
||||
- name: Verify Alloy agent is running
|
||||
systemd:
|
||||
name: alloy
|
||||
state: started
|
||||
register: alloy_status
|
||||
|
||||
- name: Display agent status
|
||||
debug:
|
||||
msg: "Monitoring agent deployed to {{ inventory_hostname }} via NetBird mesh"
|
||||
@@ -15,7 +15,14 @@
|
||||
#
|
||||
# ansible-vault edit inventory/group_vars/all/vault.yml
|
||||
# # delete vault_netbird_turn_secret, vault_netbird_relay_secret,
|
||||
# # vault_netbird_proxy_token
|
||||
# # and netbird_turn_secret
|
||||
#
|
||||
# That last one is easy to miss and breaks the rotation silently. vault.yml
|
||||
# carried BOTH vault_netbird_turn_secret and a bare netbird_turn_secret. The
|
||||
# role default is `netbird_turn_secret: "{{ vault_netbird_turn_secret }}"`, but
|
||||
# a group var outranks a role default — so the bare key wins, and rotating only
|
||||
# the vault_-prefixed one changes nothing that reaches the template. The check
|
||||
# below matches both spellings for that reason.
|
||||
#
|
||||
# The first rotation then populates netbird-rotatable.yml with fresh values, so
|
||||
# nothing needs copying across.
|
||||
@@ -29,10 +36,24 @@
|
||||
# cannot disagree across a run.
|
||||
# vault_netbird_relay_secret management.json on the server and relay.env on
|
||||
# the proxies. Two hosts, both managed here.
|
||||
# vault_netbird_proxy_token proxy.env on the server and the proxies.
|
||||
#
|
||||
# The test for "safe to rotate here" is whether the value appears in
|
||||
# management.json.j2. If the management server does not read it from a file
|
||||
# Ansible writes, Ansible does not own both ends, whatever else it templates.
|
||||
#
|
||||
# WHAT IS NOT, and must not be added:
|
||||
#
|
||||
# vault_netbird_proxy_token LOOKS symmetric — it is in proxy.env on both
|
||||
# the server and the proxies, all Ansible
|
||||
# templates — but the management server does not
|
||||
# validate it from any of them. It is registered
|
||||
# in the management DATABASE. Rotating it changes
|
||||
# only what the proxies present, so every proxy
|
||||
# is locked out: "proxy auth failed: too many
|
||||
# failed authentication attempts", and the
|
||||
# proxies stop receiving their domain
|
||||
# configuration. Learned by doing it.
|
||||
#
|
||||
# vault_netbird_datastore_key encrypts NetBird's datastore at rest. Changing
|
||||
# it without re-encrypting the store makes the
|
||||
# existing data unreadable. Rotating this needs a
|
||||
@@ -114,7 +135,7 @@
|
||||
ansible-vault view
|
||||
--vault-password-file {{ playbook_dir }}/../vault-password.sh
|
||||
{{ playbook_dir }}/../inventory/group_vars/all/vault.yml
|
||||
| grep -cE '^vault_netbird_(turn_secret|relay_secret|proxy_token):'
|
||||
| grep -cE '^(vault_)?netbird_(turn_secret|relay_secret):'
|
||||
register: _stale_keys
|
||||
changed_when: false
|
||||
# grep exits 1 on "no matches", which is the good case here.
|
||||
@@ -130,8 +151,8 @@
|
||||
rotates. group_vars files merge alphabetically and vault.yml sorts
|
||||
last, so it would override netbird-rotatable.yml on the next deploy
|
||||
and revert the secrets. Remove vault_netbird_turn_secret,
|
||||
vault_netbird_relay_secret and vault_netbird_proxy_token from
|
||||
vault.yml first (one-time migration), then re-run.
|
||||
netbird_turn_secret and vault_netbird_relay_secret from vault.yml
|
||||
first (one-time migration), then re-run.
|
||||
success_msg: vault.yml does not shadow the rotated keys.
|
||||
|
||||
tasks:
|
||||
@@ -150,7 +171,6 @@
|
||||
ansible.builtin.set_fact:
|
||||
new_turn_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
|
||||
new_relay_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
|
||||
new_proxy_token: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
|
||||
no_log: true
|
||||
|
||||
- name: Render the new values to a temporary file
|
||||
@@ -163,7 +183,6 @@
|
||||
# the next rotation overwrites this file wholesale.
|
||||
vault_netbird_turn_secret: "{{ new_turn_secret }}"
|
||||
vault_netbird_relay_secret: "{{ new_relay_secret }}"
|
||||
vault_netbird_proxy_token: "{{ new_proxy_token }}"
|
||||
no_log: true
|
||||
|
||||
# Encrypting the whole file rather than inline !vault strings: these three
|
||||
@@ -198,7 +217,6 @@
|
||||
vars:
|
||||
netbird_turn_secret: "{{ hostvars['localhost']['new_turn_secret'] }}"
|
||||
netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
|
||||
netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"
|
||||
|
||||
roles:
|
||||
- netbird-server
|
||||
@@ -213,7 +231,6 @@
|
||||
|
||||
vars:
|
||||
netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
|
||||
netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"
|
||||
|
||||
roles:
|
||||
- netbird-proxy
|
||||
@@ -227,7 +244,7 @@
|
||||
- name: Remaining manual steps
|
||||
ansible.builtin.debug:
|
||||
msg:
|
||||
- "Rotated: turn_secret, relay_secret, proxy_token."
|
||||
- "Rotated: turn_secret and relay_secret."
|
||||
- "Commit inventory/group_vars/all/netbird-rotatable.yml — it is encrypted."
|
||||
- "NOT rotated: datastore_key (needs dump/restore), auth_client_secret (Authentik side)."
|
||||
- "OpenBao is unaffected by design — its NetBird credentials are separate ones."
|
||||
|
||||
@@ -169,8 +169,7 @@ Merge the PR, then trigger the pipeline (Woodpecker → Run → `PIPELINE ==
|
||||
"deploy-claude-workstation"`), or push a change under the role's paths. CI SSHes
|
||||
to `10.3.40.60` over the tunnel and applies `common`, `netbird-client`, and this
|
||||
role. Also run the umbrella `plays/common.yml` against the new host once, so it
|
||||
picks up `qemu-guest-agent` and `monitoring-agent` (Alloy → Loki/Prometheus)
|
||||
like every other inventory host.
|
||||
picks up `qemu-guest-agent` like every other inventory host.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
# Debian 12 VM on the off-site nas-2 (Synology VMM). See README for the manual
|
||||
# bootstrap (VM creation, Netbird route, secret material) that precedes this.
|
||||
#
|
||||
# common.yml already applies `common`, `qemu-guest-agent` and `monitoring-agent`
|
||||
# (Alloy -> Loki/Prometheus) to every inventory host, so this role owns only the
|
||||
# common.yml already applies `common` and `qemu-guest-agent`
|
||||
# to every inventory host, so this role owns only the
|
||||
# Claude-specific surface: toolchain, the two credential-isolated users, the
|
||||
# read-only creds, notes sync, the unattended timers, and egress lockdown.
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
# Journal -> Loki shipping.
|
||||
#
|
||||
# The srv-network Loki (loki.srv.renner.uno:3100, no auth) is firewalled off
|
||||
# the IoT VLAN; the only endpoint these hosts can reach is the authenticated
|
||||
# The srv-network Loki (loki.srv.renner.uno:3100) no longer exists — it moved
|
||||
# into the cluster. The only endpoint these hosts can reach is the authenticated
|
||||
# cluster ingress. edge_loki_password comes from the vault.
|
||||
|
||||
edge_alloy_config_file: /etc/alloy/edge.alloy
|
||||
|
||||
@@ -7,8 +7,8 @@
|
||||
// Loki. Its own WAL/positions live on /run (tmpfs, see edge-alloy.service),
|
||||
// so this agent writes nothing to the eMMC either.
|
||||
//
|
||||
// The srv-network Loki (loki.srv.renner.uno:3100, unauthenticated) is not
|
||||
// reachable from the IoT VLAN; the only reachable endpoint is the cluster
|
||||
// The srv-network Loki (loki.srv.renner.uno:3100) no longer exists — it moved
|
||||
// into the cluster. The only reachable endpoint is the cluster
|
||||
// ingress, which enforces Traefik basic-auth — hence the credential below.
|
||||
|
||||
loki.relabel "journal" {
|
||||
|
||||
@@ -1,59 +0,0 @@
|
||||
---
|
||||
# Monitoring Agent Configuration (Grafana Alloy)
|
||||
|
||||
# Enable/disable agent deployment
|
||||
monitoring_agent_enabled: true
|
||||
|
||||
# Version and download
|
||||
alloy_version: "1.13.2"
|
||||
alloy_arch: "linux-amd64"
|
||||
alloy_download_url: "https://github.com/grafana/alloy/releases/download/v{{ alloy_version }}/alloy-{{ alloy_arch }}.zip"
|
||||
alloy_bin_path: "/usr/local/bin/alloy"
|
||||
|
||||
# Directories
|
||||
alloy_config_dir: "/etc/alloy"
|
||||
alloy_config_file: "{{ alloy_config_dir }}/config.alloy"
|
||||
alloy_data_dir: "/var/lib/alloy"
|
||||
|
||||
# Endpoints - MUST be configured in group_vars
|
||||
loki_endpoint: "http://loki.srv.renner.uno:3100"
|
||||
prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
|
||||
|
||||
# Collection settings
|
||||
alloy_collect_journal: true
|
||||
alloy_collect_syslog: true
|
||||
alloy_collect_node_metrics: true
|
||||
alloy_collect_docker: "{{ docker_socket_exists | default(false) }}"
|
||||
|
||||
# Docker container monitoring (cAdvisor)
|
||||
# Set to true on hosts that should have cAdvisor deployed
|
||||
docker_monitoring_enabled: false
|
||||
|
||||
# Scrape interval
|
||||
alloy_scrape_interval: "30s"
|
||||
|
||||
# Labels applied to all metrics/logs
|
||||
alloy_extra_labels:
|
||||
environment: "production"
|
||||
datacenter: "homelab"
|
||||
|
||||
# Systemd service
|
||||
alloy_user: "alloy"
|
||||
alloy_group: "alloy"
|
||||
|
||||
# Agent port for self-metrics
|
||||
alloy_http_listen_port: 12345
|
||||
|
||||
# Extra scrape targets for custom exporters
|
||||
# Format: list of {name: "...", address: "host:port", job: "..."}
|
||||
alloy_extra_scrape_targets: []
|
||||
|
||||
# Syslog receiver (for receiving logs from remote devices like Synology NAS)
|
||||
# Enable this on the central log collection host (e.g., loki.srv.renner.uno)
|
||||
alloy_syslog_receiver_enabled: false
|
||||
alloy_syslog_receiver_port: 1514 # Use non-privileged port, Synology can specify port
|
||||
alloy_syslog_receiver_protocol: "udp" # udp or tcp
|
||||
alloy_syslog_receiver_format: "rfc5424" # rfc3164, rfc5424, or raw (requires experimental)
|
||||
|
||||
# Stability level for experimental features (e.g., raw syslog format)
|
||||
alloy_stability_level: "generally-available" # generally-available, public-preview, experimental
|
||||
@@ -1,17 +0,0 @@
|
||||
---
|
||||
- name: Restart alloy
|
||||
systemd:
|
||||
name: alloy
|
||||
state: restarted
|
||||
daemon_reload: yes
|
||||
when: monitoring_agent_enabled | bool
|
||||
|
||||
- name: Reload systemd
|
||||
systemd:
|
||||
daemon_reload: yes
|
||||
|
||||
- name: Restart cadvisor
|
||||
systemd:
|
||||
name: cadvisor
|
||||
state: restarted
|
||||
daemon_reload: yes
|
||||
@@ -1,170 +0,0 @@
|
||||
---
|
||||
# Monitoring Agent (Grafana Alloy) Installation and Configuration
|
||||
# This role is designed to be included in common.yml for all hosts
|
||||
|
||||
- name: Skip if monitoring agent is disabled
|
||||
meta: end_host
|
||||
when: not (monitoring_agent_enabled | bool)
|
||||
|
||||
- name: Install dependencies
|
||||
apt:
|
||||
name:
|
||||
- unzip
|
||||
- curl
|
||||
state: present
|
||||
update_cache: yes
|
||||
when: ansible_facts['os_family'] == 'Debian'
|
||||
|
||||
- name: Check if Docker socket exists
|
||||
stat:
|
||||
path: /var/run/docker.sock
|
||||
register: docker_socket
|
||||
|
||||
- name: Set Docker collection fact
|
||||
set_fact:
|
||||
alloy_collect_docker: "{{ docker_socket.stat.exists }}"
|
||||
|
||||
- name: Deploy cAdvisor container for Docker metrics
|
||||
when: docker_monitoring_enabled | bool and docker_socket.stat.exists
|
||||
block:
|
||||
- name: Create cAdvisor systemd service
|
||||
copy:
|
||||
dest: /etc/systemd/system/cadvisor.service
|
||||
mode: '0644'
|
||||
content: |
|
||||
[Unit]
|
||||
Description=cAdvisor Container Monitoring
|
||||
After=docker.service
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
ExecStartPre=-/usr/bin/docker rm -f cadvisor
|
||||
ExecStart=/usr/bin/docker run --rm \
|
||||
--name=cadvisor \
|
||||
--privileged \
|
||||
--device=/dev/kmsg \
|
||||
--volume=/:/rootfs:ro \
|
||||
--volume=/var/run:/var/run:ro \
|
||||
--volume=/sys:/sys:ro \
|
||||
--volume=/var/lib/docker/:/var/lib/docker:ro \
|
||||
--volume=/dev/disk/:/dev/disk:ro \
|
||||
--volume=/cgroup:/cgroup:ro \
|
||||
--publish=127.0.0.1:8080:8080 \
|
||||
gcr.io/cadvisor/cadvisor:v0.49.1 \
|
||||
--store_container_labels=true
|
||||
ExecStop=/usr/bin/docker stop cadvisor
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
notify:
|
||||
- Reload systemd
|
||||
- Restart cadvisor
|
||||
|
||||
- name: Enable and start cAdvisor service
|
||||
systemd:
|
||||
name: cadvisor
|
||||
state: started
|
||||
enabled: yes
|
||||
daemon_reload: yes
|
||||
|
||||
- name: Create alloy user
|
||||
user:
|
||||
name: "{{ alloy_user }}"
|
||||
system: yes
|
||||
shell: /usr/sbin/nologin
|
||||
create_home: no
|
||||
groups: "{{ 'docker' if alloy_collect_docker else omit }}"
|
||||
append: yes
|
||||
|
||||
- name: Create directories
|
||||
file:
|
||||
path: "{{ item }}"
|
||||
state: directory
|
||||
owner: "{{ alloy_user }}"
|
||||
group: "{{ alloy_group }}"
|
||||
mode: '0755'
|
||||
loop:
|
||||
- "{{ alloy_config_dir }}"
|
||||
- "{{ alloy_data_dir }}"
|
||||
|
||||
- name: Check if Alloy is already installed
|
||||
stat:
|
||||
path: "{{ alloy_bin_path }}"
|
||||
register: alloy_binary
|
||||
|
||||
- name: Get installed Alloy version
|
||||
command: "{{ alloy_bin_path }} --version"
|
||||
register: installed_version
|
||||
changed_when: false
|
||||
failed_when: false
|
||||
when: alloy_binary.stat.exists
|
||||
|
||||
- name: Download Alloy
|
||||
get_url:
|
||||
url: "{{ alloy_download_url }}"
|
||||
dest: "/tmp/alloy.zip"
|
||||
mode: '0644'
|
||||
when: not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
|
||||
|
||||
- name: Unzip Alloy
|
||||
unarchive:
|
||||
src: "/tmp/alloy.zip"
|
||||
dest: "/tmp"
|
||||
remote_src: yes
|
||||
when: not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
|
||||
|
||||
- name: Find Alloy binary in extracted files
|
||||
find:
|
||||
paths: "/tmp"
|
||||
patterns: "alloy-{{ alloy_arch }}"
|
||||
file_type: file
|
||||
register: alloy_extracted_binary
|
||||
when: not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
|
||||
|
||||
- name: Install Alloy binary from extracted file
|
||||
copy:
|
||||
src: "{{ alloy_extracted_binary.files[0].path }}"
|
||||
dest: "{{ alloy_bin_path }}"
|
||||
mode: '0755'
|
||||
remote_src: yes
|
||||
when:
|
||||
- not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
|
||||
- alloy_extracted_binary.files is defined and alloy_extracted_binary.files | length > 0
|
||||
notify: Restart alloy
|
||||
|
||||
- name: Create Alloy configuration
|
||||
template:
|
||||
src: config.alloy.j2
|
||||
dest: "{{ alloy_config_file }}"
|
||||
owner: "{{ alloy_user }}"
|
||||
group: "{{ alloy_group }}"
|
||||
mode: '0644'
|
||||
notify: Restart alloy
|
||||
|
||||
- name: Create Alloy systemd service
|
||||
template:
|
||||
src: alloy.service.j2
|
||||
dest: /etc/systemd/system/alloy.service
|
||||
mode: '0644'
|
||||
notify:
|
||||
- Reload systemd
|
||||
- Restart alloy
|
||||
|
||||
- name: Enable and start Alloy service
|
||||
systemd:
|
||||
name: alloy
|
||||
state: started
|
||||
enabled: yes
|
||||
daemon_reload: yes
|
||||
|
||||
- name: Clean up downloaded files
|
||||
file:
|
||||
path: "{{ item }}"
|
||||
state: absent
|
||||
loop:
|
||||
- "/tmp/alloy.zip"
|
||||
- "/tmp/alloy-{{ alloy_arch }}"
|
||||
ignore_errors: yes
|
||||
@@ -1,37 +0,0 @@
|
||||
[Unit]
|
||||
Description=Grafana Alloy Monitoring Agent
|
||||
Documentation=https://grafana.com/docs/alloy/latest/
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User={{ alloy_user }}
|
||||
Group={{ alloy_group }}
|
||||
ExecStart={{ alloy_bin_path }} run \
|
||||
--server.http.listen-addr=0.0.0.0:{{ alloy_http_listen_port }} \
|
||||
--storage.path={{ alloy_data_dir }} \
|
||||
{% if alloy_stability_level is defined and alloy_stability_level != "generally-available" %}
|
||||
--stability.level={{ alloy_stability_level }} \
|
||||
{% endif %}
|
||||
{{ alloy_config_file }}
|
||||
ExecReload=/bin/kill -HUP $MAINPID
|
||||
Restart=on-failure
|
||||
RestartSec=10s
|
||||
LimitNOFILE=65536
|
||||
|
||||
{% if ansible_virtualization_type is defined and ansible_virtualization_type != 'lxc' %}
|
||||
# Security hardening (disabled in LXC containers due to namespace restrictions)
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ReadWritePaths={{ alloy_data_dir }} /var/log
|
||||
ReadOnlyPaths=/var/run/docker.sock
|
||||
PrivateTmp=yes
|
||||
{% endif %}
|
||||
|
||||
# Allow reading journal
|
||||
SupplementaryGroups=systemd-journal{% if alloy_collect_docker %} docker{% endif %}
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,266 +0,0 @@
|
||||
// Grafana Alloy Configuration
|
||||
// Host: {{ inventory_hostname }}
|
||||
// Generated by Ansible
|
||||
|
||||
// =====================================================
|
||||
// Logging Configuration (Loki)
|
||||
// =====================================================
|
||||
|
||||
{% if alloy_collect_journal %}
|
||||
// Collect systemd journal logs
|
||||
loki.source.journal "journal" {
|
||||
forward_to = [loki.write.default.receiver]
|
||||
labels = {
|
||||
job = "systemd-journal",
|
||||
host = "{{ inventory_hostname }}",
|
||||
{% for key, value in alloy_extra_labels.items() %}
|
||||
{{ key }} = "{{ value }}",
|
||||
{% endfor %}
|
||||
}
|
||||
relabel_rules = loki.relabel.journal.rules
|
||||
}
|
||||
|
||||
loki.relabel "journal" {
|
||||
forward_to = []
|
||||
|
||||
rule {
|
||||
source_labels = ["__journal__systemd_unit"]
|
||||
target_label = "unit"
|
||||
}
|
||||
|
||||
rule {
|
||||
source_labels = ["__journal_priority_keyword"]
|
||||
target_label = "level"
|
||||
}
|
||||
|
||||
rule {
|
||||
source_labels = ["__journal__hostname"]
|
||||
target_label = "hostname"
|
||||
}
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
{% if alloy_collect_syslog %}
|
||||
// Collect syslog files
|
||||
local.file_match "syslog" {
|
||||
path_targets = [
|
||||
{__path__ = "/var/log/syslog", job = "syslog"},
|
||||
{__path__ = "/var/log/auth.log", job = "auth"},
|
||||
{__path__ = "/var/log/kern.log", job = "kernel"},
|
||||
]
|
||||
}
|
||||
|
||||
loki.source.file "syslog" {
|
||||
targets = local.file_match.syslog.targets
|
||||
forward_to = [loki.write.default.receiver]
|
||||
tail_from_end = true
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
{% if alloy_collect_docker %}
|
||||
// Docker container logs
|
||||
discovery.docker "containers" {
|
||||
host = "unix:///var/run/docker.sock"
|
||||
}
|
||||
|
||||
discovery.relabel "docker" {
|
||||
targets = discovery.docker.containers.targets
|
||||
|
||||
rule {
|
||||
source_labels = ["__meta_docker_container_name"]
|
||||
target_label = "container"
|
||||
regex = "/(.*)"
|
||||
}
|
||||
|
||||
rule {
|
||||
source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
|
||||
target_label = "service"
|
||||
}
|
||||
|
||||
rule {
|
||||
source_labels = ["__meta_docker_container_label_com_docker_compose_project"]
|
||||
target_label = "project"
|
||||
}
|
||||
}
|
||||
|
||||
loki.source.docker "docker" {
|
||||
host = "unix:///var/run/docker.sock"
|
||||
targets = discovery.relabel.docker.output
|
||||
forward_to = [loki.process.docker.receiver]
|
||||
labels = {
|
||||
job = "docker",
|
||||
host = "{{ inventory_hostname }}",
|
||||
{% for key, value in alloy_extra_labels.items() %}
|
||||
{{ key }} = "{{ value }}",
|
||||
{% endfor %}
|
||||
}
|
||||
}
|
||||
|
||||
loki.process "docker" {
|
||||
forward_to = [loki.write.default.receiver]
|
||||
|
||||
stage.json {
|
||||
expressions = {
|
||||
log = "log",
|
||||
stream = "stream",
|
||||
time = "time",
|
||||
}
|
||||
}
|
||||
|
||||
stage.labels {
|
||||
values = {
|
||||
stream = "",
|
||||
}
|
||||
}
|
||||
|
||||
stage.output {
|
||||
source = "log"
|
||||
}
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
{% if alloy_syslog_receiver_enabled %}
|
||||
// Syslog receiver for remote devices (Synology NAS, network devices, etc.)
|
||||
loki.source.syslog "syslog_receiver" {
|
||||
listener {
|
||||
address = "0.0.0.0:{{ alloy_syslog_receiver_port }}"
|
||||
protocol = "{{ alloy_syslog_receiver_protocol }}"
|
||||
syslog_format = "{{ alloy_syslog_receiver_format }}"
|
||||
labels = {
|
||||
job = "syslog-remote",
|
||||
}
|
||||
}
|
||||
{% if alloy_syslog_receiver_format == "raw" %}
|
||||
forward_to = [loki.process.syslog_receiver.receiver]
|
||||
}
|
||||
|
||||
// Parse raw syslog messages (BSD/RFC 3164 format)
|
||||
// Synology's "IETF" format actually sends: <priority>MMM DD HH:MM:SS hostname app[pid]: message
|
||||
// Example: <14>Mar 1 09:02:47 nas-1 Connection: User [pve] from...
|
||||
loki.process "syslog_receiver" {
|
||||
forward_to = [loki.write.default.receiver]
|
||||
|
||||
stage.regex {
|
||||
expression = "^\\w+\\s+\\d+\\s+\\d+:\\d+:\\d+\\s+(?P<hostname>\\S+)\\s+(?P<app>[^:\\[]+)(?:\\[\\d+\\])?:\\s*(?P<message>.*)$"
|
||||
}
|
||||
|
||||
stage.labels {
|
||||
values = {
|
||||
hostname = "",
|
||||
app = "",
|
||||
}
|
||||
}
|
||||
|
||||
stage.output {
|
||||
source = "message"
|
||||
}
|
||||
}
|
||||
{% else %}
|
||||
forward_to = [loki.write.default.receiver]
|
||||
}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
// Write logs to Loki
|
||||
loki.write "default" {
|
||||
endpoint {
|
||||
url = "{{ loki_endpoint }}/loki/api/v1/push"
|
||||
}
|
||||
external_labels = {
|
||||
host = "{{ inventory_hostname }}",
|
||||
{% for key, value in alloy_extra_labels.items() %}
|
||||
{{ key }} = "{{ value }}",
|
||||
{% endfor %}
|
||||
}
|
||||
}
|
||||
|
||||
// =====================================================
|
||||
// Metrics Configuration (Prometheus)
|
||||
// =====================================================
|
||||
|
||||
{% if alloy_collect_node_metrics %}
|
||||
// Node metrics (CPU, memory, disk, network)
|
||||
prometheus.exporter.unix "node" {
|
||||
// Using default settings for comprehensive node metrics
|
||||
}
|
||||
|
||||
prometheus.scrape "node" {
|
||||
targets = prometheus.exporter.unix.node.targets
|
||||
forward_to = [prometheus.remote_write.default.receiver]
|
||||
scrape_interval = "{{ alloy_scrape_interval }}"
|
||||
job_name = "node"
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
{% if alloy_collect_docker and docker_monitoring_enabled %}
|
||||
// Docker/cAdvisor metrics - scrape from cAdvisor container
|
||||
prometheus.scrape "cadvisor" {
|
||||
targets = [
|
||||
{"__address__" = "127.0.0.1:8080"},
|
||||
]
|
||||
forward_to = [prometheus.relabel.cadvisor.receiver]
|
||||
scrape_interval = "{{ alloy_scrape_interval }}"
|
||||
job_name = "cadvisor"
|
||||
}
|
||||
|
||||
// Relabel cAdvisor metrics to add host label
|
||||
prometheus.relabel "cadvisor" {
|
||||
forward_to = [prometheus.remote_write.default.receiver]
|
||||
|
||||
rule {
|
||||
target_label = "host"
|
||||
replacement = "{{ inventory_hostname }}"
|
||||
}
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
// Self-monitoring metrics
|
||||
prometheus.exporter.self "alloy" {}
|
||||
|
||||
prometheus.scrape "alloy_self" {
|
||||
targets = prometheus.exporter.self.alloy.targets
|
||||
forward_to = [prometheus.remote_write.default.receiver]
|
||||
scrape_interval = "{{ alloy_scrape_interval }}"
|
||||
job_name = "alloy"
|
||||
}
|
||||
|
||||
{% if alloy_extra_scrape_targets | length > 0 %}
|
||||
// Extra scrape targets (custom exporters)
|
||||
{% for target in alloy_extra_scrape_targets %}
|
||||
prometheus.scrape "{{ target.name }}" {
|
||||
targets = [
|
||||
{"__address__" = "{{ target.address }}"},
|
||||
]
|
||||
forward_to = [prometheus.relabel.extra_{{ target.name }}.receiver]
|
||||
scrape_interval = "{{ alloy_scrape_interval }}"
|
||||
job_name = "{{ target.job }}"
|
||||
}
|
||||
|
||||
prometheus.relabel "extra_{{ target.name }}" {
|
||||
forward_to = [prometheus.remote_write.default.receiver]
|
||||
|
||||
rule {
|
||||
target_label = "host"
|
||||
replacement = "{{ inventory_hostname }}"
|
||||
}
|
||||
}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
|
||||
// Remote write to Prometheus
|
||||
prometheus.remote_write "default" {
|
||||
endpoint {
|
||||
url = "{{ prometheus_remote_write_endpoint }}"
|
||||
}
|
||||
external_labels = {
|
||||
host = "{{ inventory_hostname }}",
|
||||
{% for key, value in alloy_extra_labels.items() %}
|
||||
{{ key }} = "{{ value }}",
|
||||
{% endfor %}
|
||||
}
|
||||
}
|
||||
|
||||
// =====================================================
|
||||
// Agent HTTP Server (for health checks and metrics)
|
||||
// =====================================================
|
||||
// Agent serves metrics on port {{ alloy_http_listen_port }}
|
||||
@@ -31,8 +31,8 @@ all in state that outlived the VM.
|
||||
two hosts every PG is undersized with min_size 2, so one further OSD loss
|
||||
blocks IO on every CephFS mount cluster-wide.
|
||||
- [ ] Talos metal ISO on a USB stick. Schematic
|
||||
`a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
|
||||
(`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
|
||||
`0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9`
|
||||
(`net.ifnames=1`, intel-ucode, thunderbolt, i915; no qemu-guest-agent).
|
||||
Verify by comparing partition offsets and sizes against the ISO with
|
||||
`fdisk -l` — blkid labels lie after a rewrite.
|
||||
- [ ] Physical access. Secure Boot must be turned off (see below), and that is
|
||||
@@ -383,6 +383,34 @@ raised to `high_recovery_ops` to speed the evacuation.
|
||||
The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
|
||||
`ceph config-key rm device/<devid>`; the mgr's devicehealth view may lag.
|
||||
|
||||
### Long-lived TCP sessions to devices outside the cluster die silently
|
||||
|
||||
Check this before declaring the conversion done. Anything holding a persistent
|
||||
connection to a device on another VLAN — the ESPHome integration is the case
|
||||
that bit us — keeps a half-open socket when the node underneath it moves. The
|
||||
peer drops its side; the pod's side never learns, because no RST arrives and no
|
||||
keepalive fires.
|
||||
|
||||
The symptom is that there is no symptom. Home Assistant logged nothing, retried
|
||||
nothing, and reported the devices as connected. The devices themselves showed
|
||||
"not connected". Hours went into reading logs that were empty by construction.
|
||||
|
||||
`/proc/net/tcp` in the pod settles it in one command — `01` is ESTABLISHED,
|
||||
`02` is SYN_SENT, and the port is hex:
|
||||
|
||||
```sh
|
||||
kubectl -n home-assistant exec <ha-pod> -c home-assistant -- \
|
||||
sh -c 'grep ":17A5" /proc/net/tcp' # 17A5 = 6053, the ESPHome API
|
||||
```
|
||||
|
||||
Seven `01` entries to devices insisting they were disconnected is the tell. A
|
||||
`02` alongside them is a genuinely dead device, not a stale socket — that one
|
||||
Home Assistant *was* retrying and *did* log.
|
||||
|
||||
Restarting the consuming pod clears it. Reloading just the integration from the
|
||||
UI does the same with less disruption. Neither is discoverable from logs, which
|
||||
is why it belongs here.
|
||||
|
||||
## Known-stale things that will bite
|
||||
|
||||
- `bootstrap.sh` cannot generate a config at all right now (issue #581), so a
|
||||
|
||||
@@ -87,7 +87,7 @@ all three nodes identically from there, since `apply-config --insecure` does
|
||||
not care whether the target is a VM or bare metal.
|
||||
|
||||
```
|
||||
https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
|
||||
https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.12.11/metal-amd64.iso
|
||||
```
|
||||
|
||||
That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
|
||||
|
||||
@@ -37,6 +37,13 @@ resources:
|
||||
- esphome-service.yaml
|
||||
- esphome-gate-service.yaml
|
||||
- esphome-ingress.yaml
|
||||
# mDNS relay — carries IoT-VLAN announcements into HA's zeroconf listener as
|
||||
# unicast, which is the only direction the CNI will move them. Without it HA
|
||||
# never learns that a device's DHCP lease moved. See the header comment in
|
||||
# mdns-relay-deployment.yaml.
|
||||
- mdns-relay-network-attachment.yaml
|
||||
- mdns-relay-service.yaml
|
||||
- mdns-relay-deployment.yaml
|
||||
# Zigbee2MQTT — ZHA replacement bridge. Ships scaled to 0; the cutover
|
||||
# runbook (zigbee2mqtt-migration-runbook.md) scales it up. The MQTT
|
||||
# `zigbee2mqtt` broker user is provisioned by mosquitto-bootstrap.yaml
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
---
|
||||
# mDNS relay: gives Home Assistant sight of the IoT VLAN's mDNS announcements
|
||||
# without putting a macvlan on the HA pod itself.
|
||||
#
|
||||
# THE PROBLEM
|
||||
#
|
||||
# HA's zeroconf listener joins both mDNS groups on eth0 — 224.0.0.251 and
|
||||
# ff02::fb — and never hears anything on either.
|
||||
# Multicast is link-local, so IoT-VLAN announcements do not reach the pod
|
||||
# CIDR; and Cilium runs in tunnel mode (VXLAN) without `enable-multicast`, so
|
||||
# it does not replicate multicast between pods either. Both halves of the path
|
||||
# are dead, which is why an avahi-style reflector — receive on one interface,
|
||||
# re-emit as multicast on the other — cannot work here: the re-emitted packets
|
||||
# would be dropped by the CNI.
|
||||
#
|
||||
# Consequence for HA: the ESPHome integration dials devices at the address
|
||||
# cached in its config entry. When a lease rotates, every attempt fails with
|
||||
# `[Errno 113] Connect call failed` and the device stays unreachable, because
|
||||
# the mDNS announcement that would carry the new address never arrives. New
|
||||
# devices are never discovered at all.
|
||||
#
|
||||
# THE MECHANISM
|
||||
#
|
||||
# This relay holds the multicast membership on the IoT VLAN through its own
|
||||
# macvlan (mdns-relay-network-attachment.yaml) and forwards each datagram it
|
||||
# receives to HA as ordinary UNICAST UDP. That is the one direction of travel
|
||||
# the CNI will carry.
|
||||
#
|
||||
# It works because python-zeroconf does not filter on how a packet arrived:
|
||||
# its datagram handler validates size, deduplicates, and parses, but performs
|
||||
# no check on destination address, source address, or source port. A unicast
|
||||
# datagram delivered to :5353 is processed exactly like a multicast one. (mDNS
|
||||
# unicast responses are part of RFC 6762 — the QU bit — so this is within the
|
||||
# protocol, not a trick.)
|
||||
#
|
||||
# WHAT THIS DOES AND DOES NOT RESTORE
|
||||
#
|
||||
# Restored: passive reception. RFC 6762 requires a responder to re-announce
|
||||
# when its address changes, so a device that moves to a new lease announces
|
||||
# the move, HA sees it, and the ESPHome integration updates the config entry.
|
||||
# That is the failure this exists to fix. New devices announcing themselves
|
||||
# are likewise seen.
|
||||
#
|
||||
# NOT restored: HA-initiated browsing. Queries HA sends leave on eth0 as
|
||||
# multicast and die there, so HA cannot actively solicit a response. Anything
|
||||
# depending on a query rather than an announcement stays broken. Making that
|
||||
# work needs a querier on the VLAN side, which this relay deliberately is not.
|
||||
#
|
||||
# EXPOSURE
|
||||
#
|
||||
# The receive sockets carry so-bindtodevice=net1, so only datagrams that
|
||||
# actually arrived on the IoT VLAN interface are delivered. Binding to net1's
|
||||
# address instead is not possible — it is DHCP-leased and unknown at exec time
|
||||
# — but binding to the device achieves the same confinement without needing to
|
||||
# know the address, and needs no capability: SO_BINDTODEVICE succeeds as an
|
||||
# unprivileged uid here.
|
||||
#
|
||||
# Without it socat would bind the wildcard, and since the namespace CNP grants
|
||||
# every pod in home-assistant unrestricted access to every other pod on every
|
||||
# port, any sibling pod could reach :5353 over the cluster network and have
|
||||
# records forwarded into HA's zeroconf cache.
|
||||
#
|
||||
# What remains, and cannot be closed here: any host on the IoT VLAN can send
|
||||
# this port a datagram. That is the same trust already extended to genuine
|
||||
# mDNS from that segment, which any device there can equally forge — the relay
|
||||
# widens who can reach HA's listener, not what can be claimed. macvlan traffic
|
||||
# bypasses Cilium policy entirely, so the namespace CNP does not constrain
|
||||
# this interface either way.
|
||||
#
|
||||
# Note that the relay is not what makes HA's zeroconf reachable from inside
|
||||
# the cluster: HA binds :5353 on the pod address (v4) and the wildcard (v6),
|
||||
# and the namespace CNP already permits any sibling pod to reach it directly.
|
||||
# Confining the relay removes a second route to that listener; it does not
|
||||
# remove the first.
|
||||
#
|
||||
# Nothing else is exposed: the containers run socat and only socat, with no
|
||||
# second listener to reach.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: mdns-relay
|
||||
namespace: home-assistant
|
||||
labels:
|
||||
app.kubernetes.io/name: mdns-relay
|
||||
spec:
|
||||
# One replica. Extra replicas would deliver a duplicate of every announcement
|
||||
# — harmless, since zeroconf deduplicates, but they buy nothing: this is a
|
||||
# cache-refresh path, and a missed window is corrected by the next
|
||||
# announcement.
|
||||
replicas: 1
|
||||
# Recreate, not RollingUpdate: two relays would briefly hold two DHCP leases
|
||||
# on v30iot for one workload.
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mdns-relay
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: mdns-relay
|
||||
annotations:
|
||||
k8s.v1.cni.cncf.io/networks: mdns-relay-iot
|
||||
spec:
|
||||
enableServiceLinks: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65534
|
||||
runAsGroup: 65534
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
# One container per address family. mDNS is two protocols — 224.0.0.251
|
||||
# and ff02::fb — and a socket joined to one hears nothing of the other.
|
||||
# HA listens on both (its zeroconf holds both groups on eth0), so a
|
||||
# v4-only relay would silently drop every IPv6 announcement, and devices
|
||||
# that announce only over v6 would stay invisible exactly as before.
|
||||
#
|
||||
# Two processes rather than one: socat carries a single address pair per
|
||||
# invocation, and the families need different receive and send addresses.
|
||||
# They are independent — one crashing does not blind the other — and each
|
||||
# is restarted by the kubelet on its own.
|
||||
containers:
|
||||
# -u is load-bearing on both: unidirectional, left to right. Without it
|
||||
# socat would also carry HA's replies back out, and this must stay a
|
||||
# one-way tap.
|
||||
#
|
||||
# UDP{4,6}-RECV (not -RECVFROM) keeps reading from one socket instead of
|
||||
# forking per datagram — the IoT VLAN's mDNS chatter is continuous, and
|
||||
# a fork per packet would be a process storm.
|
||||
#
|
||||
# The group is joined on net1 specifically, so the relay subscribes on
|
||||
# the IoT VLAN rather than on the pod network.
|
||||
#
|
||||
# Both send targets are the same ClusterIP name, resolved once at start;
|
||||
# UDP4-SENDTO takes its A record and UDP6-SENDTO its AAAA. See
|
||||
# mdns-relay-service.yaml for why that indirection is required and why
|
||||
# the Service is dual-stack.
|
||||
- name: socat-ipv4
|
||||
image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- -u
|
||||
- "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr"
|
||||
- "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
|
||||
# No probes. socat is a single process with no health surface: if it
|
||||
# exits, the container exits and the kubelet restarts it, which is the
|
||||
# entire liveness story. A probe here could only re-assert that PID 1
|
||||
# is running.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 64Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
# Joining ff02::fb needs no address configuration on net1: the group is
|
||||
# link-local and the kernel's automatic link-local address on the
|
||||
# macvlan is sufficient to receive on it. Whether v30iot hands out
|
||||
# global v6 addresses does not affect reception here.
|
||||
#
|
||||
# ipv6-v6only=1 keeps this socket to v6 only. Both containers share the
|
||||
# pod's network namespace, and a dual-stack v6 socket would overlap the
|
||||
# v4 container's port. Measured behaviour without it was no contention —
|
||||
# the v4 receiver kept taking packets while the v6 one ran — but that
|
||||
# depends on the kernel's bindv6only default rather than on anything
|
||||
# stated here, so the roles are made explicit instead of inherited.
|
||||
- name: socat-ipv6
|
||||
image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- -u
|
||||
- "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr"
|
||||
- "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 64Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
# mdns-relay's second NIC: macvlan onto v30iot, so the relay has real L2
|
||||
# presence on the IoT VLAN and can join both mDNS multicast groups
|
||||
# (224.0.0.251 and ff02::fb, port 5353). The v6 group needs no address
|
||||
# configuration of its own: it is link-local, and the kernel's automatic
|
||||
# link-local address on the macvlan is enough to receive on it, independently
|
||||
# of whether v30iot hands out global v6 addresses.
|
||||
# Multicast is link-local — it does not cross the L3
|
||||
# boundary between the cluster pod CIDR and v30iot — so that membership has to
|
||||
# be held by a pod sitting physically on that segment.
|
||||
#
|
||||
# The address is leased from the UniFi DHCP server on v30iot. Nothing
|
||||
# references it: the relay only receives on this interface and sends onward
|
||||
# over eth0. See mdns-relay-deployment.yaml for what the relay does with it.
|
||||
#
|
||||
# Multus's `master` is an interface name. On the k8s-{1,2,3} nodes the v30iot
|
||||
# NIC is `ens19` — a stable, busPath-derived predictable name (net.ifnames=1,
|
||||
# via the Image Factory schematic), identical across nodes and independent of
|
||||
# driver probe order. This is the same master shelly-manager-iot uses.
|
||||
apiVersion: k8s.cni.cncf.io/v1
|
||||
kind: NetworkAttachmentDefinition
|
||||
metadata:
|
||||
name: mdns-relay-iot
|
||||
namespace: home-assistant
|
||||
spec:
|
||||
# Chained CNI, identical in shape to shelly-manager-iot: macvlan leases the
|
||||
# v30iot address by DHCP, then `sbr` confines net1's routes to a
|
||||
# source-based routing table. Without sbr the DHCP-supplied default gateway
|
||||
# (10.2.30.1, option 3) installs a second default route at the same metric
|
||||
# as eth0's, and cluster-internal traffic egresses the IoT VLAN and is
|
||||
# dropped. The relay's forward path to Home Assistant runs over eth0, so
|
||||
# that route has to stay intact.
|
||||
#
|
||||
# Requires the cni-dhcp-daemon DaemonSet in infrastructure/multus.
|
||||
config: |
|
||||
{
|
||||
"cniVersion": "0.3.1",
|
||||
"name": "mdns-relay-iot",
|
||||
"plugins": [
|
||||
{
|
||||
"type": "macvlan",
|
||||
"master": "ens19",
|
||||
"mode": "bridge",
|
||||
"ipam": {
|
||||
"type": "dhcp"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "sbr"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
# Stable delivery target for the relay's forwarded mDNS traffic.
|
||||
#
|
||||
# The relay resolves its send target once, at process start (socat has no
|
||||
# re-resolution loop), so it cannot chase the HA pod IP across restarts. A
|
||||
# ClusterIP is the fix: the IP is stable for the lifetime of the Service, and
|
||||
# Cilium re-DNATs to whatever pod currently backs it.
|
||||
#
|
||||
# Deliberately a separate Service from `home-assistant`, which targets the
|
||||
# nginx proxy sidecar on 8124 and is the XFF trust anchor for the public UI.
|
||||
# This one bypasses the sidecar and addresses the home-assistant container's
|
||||
# own port directly — containers in a pod share a network namespace, so
|
||||
# targetPort 5353 lands on python-zeroconf's listener regardless of which
|
||||
# container opened it. Keeping the two apart means the UI path and the
|
||||
# discovery path cannot be changed by accident through one another.
|
||||
#
|
||||
# No CiliumNetworkPolicy change is needed: home-assistant-ns.yaml already
|
||||
# permits intra-namespace traffic on all ports, and the relay is in this
|
||||
# namespace.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: home-assistant-mdns
|
||||
namespace: home-assistant
|
||||
labels:
|
||||
app.kubernetes.io/name: home-assistant
|
||||
spec:
|
||||
# Dual-stack, because mDNS is two protocols and HA listens on both: its
|
||||
# zeroconf socket holds 224.0.0.251 on eth0 and ff02::fb alongside it. The
|
||||
# relay forwards each family over its own transport rather than folding v6
|
||||
# announcements into a v4 hop, so the Service has to answer on both.
|
||||
ipFamilyPolicy: PreferDualStack
|
||||
ipFamilies:
|
||||
- IPv4
|
||||
- IPv6
|
||||
selector:
|
||||
app.kubernetes.io/name: home-assistant
|
||||
ports:
|
||||
- name: mdns
|
||||
port: 5353
|
||||
protocol: UDP
|
||||
targetPort: 5353
|
||||
@@ -4,3 +4,12 @@ resources:
|
||||
- cni-plugins-installer.yaml
|
||||
- cni-dhcp-daemon.yaml
|
||||
- multus-daemonset.yaml
|
||||
# Whereabouts — cluster-coordinated IPAM, for attachments on segments with
|
||||
# no DHCP server. CRDs first: the DaemonSet's control loop reads them at
|
||||
# startup. See the header in whereabouts-daemonset.yaml for why neither
|
||||
# dhcp nor host-local covers the Ceph SAN.
|
||||
- whereabouts-crd-ippools.yaml
|
||||
- whereabouts-crd-overlappingrangeipreservations.yaml
|
||||
- whereabouts-crd-nodeslicepools.yaml
|
||||
- whereabouts-daemonset.yaml
|
||||
- whereabouts-reconciler.yaml
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
apiVersion: apiextensions.k8s.io/v1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
annotations:
|
||||
controller-gen.kubebuilder.io/version: v0.14.0
|
||||
name: ippools.whereabouts.cni.cncf.io
|
||||
spec:
|
||||
group: whereabouts.cni.cncf.io
|
||||
names:
|
||||
kind: IPPool
|
||||
listKind: IPPoolList
|
||||
plural: ippools
|
||||
singular: ippool
|
||||
scope: Namespaced
|
||||
versions:
|
||||
- name: v1alpha1
|
||||
schema:
|
||||
openAPIV3Schema:
|
||||
description: IPPool is the Schema for the ippools API
|
||||
properties:
|
||||
apiVersion:
|
||||
description: |-
|
||||
APIVersion defines the versioned schema of this representation of an object.
|
||||
Servers should convert recognized schemas to the latest internal value, and
|
||||
may reject unrecognized values.
|
||||
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
|
||||
type: string
|
||||
kind:
|
||||
description: |-
|
||||
Kind is a string value representing the REST resource this object represents.
|
||||
Servers may infer this from the endpoint the client submits requests to.
|
||||
Cannot be updated.
|
||||
In CamelCase.
|
||||
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
|
||||
type: string
|
||||
metadata:
|
||||
type: object
|
||||
spec:
|
||||
description: IPPoolSpec defines the desired state of IPPool
|
||||
properties:
|
||||
allocations:
|
||||
additionalProperties:
|
||||
description: IPAllocation represents metadata about the pod/container
|
||||
owner of a specific IP
|
||||
properties:
|
||||
id:
|
||||
type: string
|
||||
ifname:
|
||||
type: string
|
||||
podref:
|
||||
type: string
|
||||
required:
|
||||
- id
|
||||
- podref
|
||||
type: object
|
||||
description: |-
|
||||
Allocations is the set of allocated IPs for the given range. Its` indices are a direct mapping to the
|
||||
IP with the same index/offset for the pool's range.
|
||||
type: object
|
||||
range:
|
||||
description: Range is a RFC 4632/4291-style string that represents
|
||||
an IP address and prefix length in CIDR notation
|
||||
type: string
|
||||
required:
|
||||
- allocations
|
||||
- range
|
||||
type: object
|
||||
type: object
|
||||
served: true
|
||||
storage: true
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
apiVersion: apiextensions.k8s.io/v1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
annotations:
|
||||
controller-gen.kubebuilder.io/version: v0.14.0
|
||||
name: nodeslicepools.whereabouts.cni.cncf.io
|
||||
spec:
|
||||
group: whereabouts.cni.cncf.io
|
||||
names:
|
||||
kind: NodeSlicePool
|
||||
listKind: NodeSlicePoolList
|
||||
plural: nodeslicepools
|
||||
singular: nodeslicepool
|
||||
scope: Namespaced
|
||||
versions:
|
||||
- name: v1alpha1
|
||||
schema:
|
||||
openAPIV3Schema:
|
||||
description: NodeSlicePool is the Schema for the nodesliceippools API
|
||||
properties:
|
||||
apiVersion:
|
||||
description: |-
|
||||
APIVersion defines the versioned schema of this representation of an object.
|
||||
Servers should convert recognized schemas to the latest internal value, and
|
||||
may reject unrecognized values.
|
||||
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
|
||||
type: string
|
||||
kind:
|
||||
description: |-
|
||||
Kind is a string value representing the REST resource this object represents.
|
||||
Servers may infer this from the endpoint the client submits requests to.
|
||||
Cannot be updated.
|
||||
In CamelCase.
|
||||
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
|
||||
type: string
|
||||
metadata:
|
||||
type: object
|
||||
spec:
|
||||
description: NodeSlicePoolSpec defines the desired state of NodeSlicePool
|
||||
properties:
|
||||
range:
|
||||
description: |-
|
||||
Range is a RFC 4632/4291-style string that represents an IP address and prefix length in CIDR notation
|
||||
this refers to the entire range where the node is allocated a subset
|
||||
type: string
|
||||
sliceSize:
|
||||
description: SliceSize is the size of subnets or slices of the range
|
||||
that each node will be assigned
|
||||
type: string
|
||||
required:
|
||||
- range
|
||||
- sliceSize
|
||||
type: object
|
||||
status:
|
||||
description: NodeSlicePoolStatus defines the desired state of NodeSlicePool
|
||||
properties:
|
||||
allocations:
|
||||
description: Allocations holds the allocations of nodes to slices
|
||||
items:
|
||||
properties:
|
||||
nodeName:
|
||||
description: NodeName is the name of the node assigned to this
|
||||
slice, empty node name is an available slice for assignment
|
||||
type: string
|
||||
sliceRange:
|
||||
description: SliceRange is the subnet of this slice
|
||||
type: string
|
||||
required:
|
||||
- nodeName
|
||||
- sliceRange
|
||||
type: object
|
||||
type: array
|
||||
required:
|
||||
- allocations
|
||||
type: object
|
||||
type: object
|
||||
served: true
|
||||
storage: true
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
apiVersion: apiextensions.k8s.io/v1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
annotations:
|
||||
controller-gen.kubebuilder.io/version: v0.14.0
|
||||
name: overlappingrangeipreservations.whereabouts.cni.cncf.io
|
||||
spec:
|
||||
group: whereabouts.cni.cncf.io
|
||||
names:
|
||||
kind: OverlappingRangeIPReservation
|
||||
listKind: OverlappingRangeIPReservationList
|
||||
plural: overlappingrangeipreservations
|
||||
singular: overlappingrangeipreservation
|
||||
scope: Namespaced
|
||||
versions:
|
||||
- name: v1alpha1
|
||||
schema:
|
||||
openAPIV3Schema:
|
||||
description: OverlappingRangeIPReservation is the Schema for the OverlappingRangeIPReservations
|
||||
API
|
||||
properties:
|
||||
apiVersion:
|
||||
description: |-
|
||||
APIVersion defines the versioned schema of this representation of an object.
|
||||
Servers should convert recognized schemas to the latest internal value, and
|
||||
may reject unrecognized values.
|
||||
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
|
||||
type: string
|
||||
kind:
|
||||
description: |-
|
||||
Kind is a string value representing the REST resource this object represents.
|
||||
Servers may infer this from the endpoint the client submits requests to.
|
||||
Cannot be updated.
|
||||
In CamelCase.
|
||||
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
|
||||
type: string
|
||||
metadata:
|
||||
type: object
|
||||
spec:
|
||||
description: OverlappingRangeIPReservationSpec defines the desired state
|
||||
of OverlappingRangeIPReservation
|
||||
properties:
|
||||
containerid:
|
||||
type: string
|
||||
ifname:
|
||||
type: string
|
||||
podref:
|
||||
type: string
|
||||
required:
|
||||
- podref
|
||||
type: object
|
||||
required:
|
||||
- spec
|
||||
type: object
|
||||
served: true
|
||||
storage: true
|
||||
@@ -0,0 +1,196 @@
|
||||
---
|
||||
# Whereabouts: cluster-wide IPAM for Multus attachments.
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
#
|
||||
# The attachments in this cluster lease their addresses by DHCP
|
||||
# (cni-dhcp-daemon.yaml). That works where a DHCP server is on the wire —
|
||||
# v30iot has one. It does not work on the Ceph SAN (VLAN 49), which carries
|
||||
# Ceph-internal traffic and nothing else, by design: there is no DHCP server
|
||||
# there and there must not be one.
|
||||
#
|
||||
# `host-local`, already installed by cni-plugins-installer.yaml, cannot stand
|
||||
# in. It keeps its allocation state per node with no cluster-wide
|
||||
# coordination, so on a shared L2 segment two nodes hand out the same address.
|
||||
# Whereabouts coordinates through the Kubernetes API instead — allocations
|
||||
# live in the ippools/overlappingrangeipreservations CRDs, so nothing about
|
||||
# the address assignment touches the segment being addressed. That property is
|
||||
# what makes it usable on an isolated SAN.
|
||||
#
|
||||
# HOST ACCESS
|
||||
#
|
||||
# The DaemonSet runs with hostNetwork and privileged. That is how upstream
|
||||
# ships it, and it is what the three CNI components already here do —
|
||||
# multus-daemonset.yaml, cni-dhcp-daemon.yaml and cni-plugins-installer.yaml
|
||||
# all carry both. The reason is the same in each case: the pod writes a CNI
|
||||
# binary and its config into /opt/cni/bin and /etc/cni/net.d on the node,
|
||||
# which is node plumbing rather than a workload, and it must work before the
|
||||
# attachment it provides addresses for exists.
|
||||
#
|
||||
# The reconciler is deliberately NOT in here. It needs neither, and runs as an
|
||||
# ordinary pod-network Deployment — see whereabouts-reconciler.yaml.
|
||||
#
|
||||
# Vendored from upstream v0.9.4 (doc/crds/daemonset-install.yaml), with the
|
||||
# image tag pinned; upstream ships `:latest`. The CRDs it needs are the three
|
||||
# whereabouts-crd-*.yaml files alongside this one, vendored unmodified so an
|
||||
# upstream diff stays readable.
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: whereabouts
|
||||
namespace: kube-system
|
||||
---
|
||||
kind: ClusterRoleBinding
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
metadata:
|
||||
name: whereabouts
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: whereabouts-cni
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: whereabouts
|
||||
namespace: kube-system
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: whereabouts-cni
|
||||
rules:
|
||||
- apiGroups:
|
||||
- whereabouts.cni.cncf.io
|
||||
resources:
|
||||
- ippools
|
||||
- overlappingrangeipreservations
|
||||
- nodeslicepools
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
- create
|
||||
- update
|
||||
- patch
|
||||
- delete
|
||||
- apiGroups:
|
||||
- coordination.k8s.io
|
||||
resources:
|
||||
- leases
|
||||
verbs:
|
||||
- '*'
|
||||
- apiGroups: [""]
|
||||
resources:
|
||||
- pods
|
||||
verbs:
|
||||
- list
|
||||
- watch
|
||||
- get
|
||||
- apiGroups: [""]
|
||||
resources:
|
||||
- nodes
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
- apiGroups: ["k8s.cni.cncf.io"]
|
||||
resources:
|
||||
- network-attachment-definitions
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
- apiGroups:
|
||||
- ""
|
||||
- events.k8s.io
|
||||
resources:
|
||||
- events
|
||||
verbs:
|
||||
- create
|
||||
- patch
|
||||
- update
|
||||
- get
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: whereabouts-config
|
||||
namespace: kube-system
|
||||
annotations:
|
||||
kubernetes.io/description: |
|
||||
Configmap containing user customizable cronjob schedule
|
||||
data:
|
||||
# Read by the reconciler (whereabouts-reconciler.yaml), not by this
|
||||
# DaemonSet. Governs how often stale allocations are swept — the case that
|
||||
# matters here is a node lost hard enough that the CNI DEL never ran, which
|
||||
# otherwise leaves an address held by a pod that no longer exists.
|
||||
cron-expression: "30 4 * * *"
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: whereabouts
|
||||
namespace: kube-system
|
||||
labels:
|
||||
tier: node
|
||||
app: whereabouts
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
name: whereabouts
|
||||
updateStrategy:
|
||||
type: RollingUpdate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
tier: node
|
||||
app: whereabouts
|
||||
name: whereabouts
|
||||
spec:
|
||||
hostNetwork: true
|
||||
serviceAccountName: whereabouts
|
||||
# Every node here is a control plane node, so the NoSchedule toleration
|
||||
# is not optional — without it this schedules nowhere.
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
effect: NoSchedule
|
||||
containers:
|
||||
- name: whereabouts
|
||||
command: ["/bin/sh"]
|
||||
args:
|
||||
- -c
|
||||
- |
|
||||
SLEEP=false source /install-cni.sh
|
||||
/token-watcher.sh &
|
||||
/ip-control-loop -log-level debug
|
||||
image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
|
||||
env:
|
||||
- name: NODENAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
apiVersion: v1
|
||||
fieldPath: spec.nodeName
|
||||
- name: WHEREABOUTS_NAMESPACE
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.namespace
|
||||
resources:
|
||||
requests:
|
||||
cpu: "100m"
|
||||
memory: "100Mi"
|
||||
limits:
|
||||
cpu: "100m"
|
||||
memory: "200Mi"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- name: cnibin
|
||||
mountPath: /host/opt/cni/bin
|
||||
- name: cni-net-dir
|
||||
mountPath: /host/etc/cni/net.d
|
||||
volumes:
|
||||
- name: cnibin
|
||||
hostPath:
|
||||
path: /opt/cni/bin
|
||||
- name: cni-net-dir
|
||||
hostPath:
|
||||
path: /etc/cni/net.d
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
# Sweeps whereabouts allocations that no live pod holds any more.
|
||||
#
|
||||
# The normal release path is the CNI DEL when a pod goes away. It does not run
|
||||
# when a node is lost hard — power cut, kernel panic, a reboot that never
|
||||
# comes back — and the allocation then stays claimed by a pod that no longer
|
||||
# exists. On a small range that matters: the Ceph SAN is a /29, so a handful
|
||||
# of leaked addresses is the difference between an OSD getting an address and
|
||||
# not.
|
||||
#
|
||||
# Runs as an ordinary pod-network Deployment. Unlike the DaemonSet next to it,
|
||||
# this needs no host access at all — it only talks to the Kubernetes API,
|
||||
# where the allocations live.
|
||||
#
|
||||
# replicas stays at 1: there is no leader election, so a second replica would
|
||||
# reconcile the same records concurrently.
|
||||
#
|
||||
# Vendored from upstream v0.9.4 (doc/crds/reconciler-deployment.yaml) with the
|
||||
# image tag pinned; upstream ships `:latest`. The schedule comes from the
|
||||
# whereabouts-config ConfigMap in whereabouts-daemonset.yaml.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: whereabouts-reconciler
|
||||
namespace: kube-system
|
||||
labels:
|
||||
tier: node
|
||||
app: whereabouts-reconciler
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
name: whereabouts-reconciler
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
tier: node
|
||||
app: whereabouts-reconciler
|
||||
name: whereabouts-reconciler
|
||||
spec:
|
||||
serviceAccountName: whereabouts
|
||||
# Every node here is a control plane node.
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
effect: NoSchedule
|
||||
containers:
|
||||
- name: whereabouts-reconciler
|
||||
command: ["/ip-reconciler"]
|
||||
args:
|
||||
- -log-level
|
||||
- debug
|
||||
image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
|
||||
env:
|
||||
- name: WHEREABOUTS_NAMESPACE
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.namespace
|
||||
resources:
|
||||
requests:
|
||||
cpu: "100m"
|
||||
memory: "50Mi"
|
||||
limits:
|
||||
cpu: "100m"
|
||||
memory: "50Mi"
|
||||
volumeMounts:
|
||||
- name: cron-scheduler-configmap
|
||||
mountPath: /cron-schedule
|
||||
volumes:
|
||||
- name: cron-scheduler-configmap
|
||||
configMap:
|
||||
name: whereabouts-config
|
||||
defaultMode: 0744
|
||||
items:
|
||||
- key: "cron-expression"
|
||||
path: "config"
|
||||
@@ -20,16 +20,30 @@ machine:
|
||||
# the MS-01 mesh later needs neither a new image
|
||||
# nor a reinstall — which is exactly the property
|
||||
# lost by installing without it.
|
||||
# siderolabs/i915 Intel GPU microcode and kernel modules for the
|
||||
# Alder Lake-P Iris Xe in the MS-01. No workload
|
||||
# requests /dev/dri today; it is here for the same
|
||||
# reason as thunderbolt, so that wanting hardware
|
||||
# transcoding later is a config change rather than
|
||||
# another rolling upgrade of every node.
|
||||
#
|
||||
# a21cec8f… is the bare-metal schematic described in RUNBOOK-cold-rebuild.md:
|
||||
# same net.ifnames=1 as the Proxmox one (the predictable interface names the
|
||||
# per-node VLAN parents match against), minus qemu-guest-agent.
|
||||
# Not included, and why, so the absences are not mistaken for oversights:
|
||||
# the X710 (i40e) and I226 (igc) NICs need no firmware extension —
|
||||
# intel-ice-firmware covers E810/ice only. mei is a prerequisite for Intel
|
||||
# Arc discrete GPUs, of which there are none. iscsi-tools, nvme-cli and
|
||||
# util-linux-tools have no consumer: storage is Ceph RBD, and device health
|
||||
# is reported by Ceph itself rather than a node-level exporter.
|
||||
#
|
||||
# 0f6fb8bd… is the bare-metal schematic: same net.ifnames=1 as the Proxmox
|
||||
# one (the predictable interface names the per-node VLAN parents match
|
||||
# against), minus qemu-guest-agent. It supersedes a21cec8f…, which was
|
||||
# identical but without i915 — see RUNBOOK-cold-rebuild.md.
|
||||
#
|
||||
# Only read at install and upgrade time. Changing it here does nothing to a
|
||||
# running node until that node is upgraded. Bump the tag with the Talos
|
||||
# version — it must match the running version, or the upgrade is a downgrade.
|
||||
install:
|
||||
image: factory.talos.dev/metal-installer/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64:v1.12.6
|
||||
image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.12.11
|
||||
features:
|
||||
rbac: true
|
||||
stableHostname: true
|
||||
|
||||
Reference in New Issue
Block a user