chore(rook-ceph): return mon_data_avail_warn to the Ceph default

The override existed because the mons share the node's /var with containerd, and on the VMs that was ~60 GB — 30% free measured image and snapshot churn rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the image cache filled and GC could not reclaim it (#434). Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so 30% free is a real reserve again and the ~106 MB mon database is irrelevant against it. Reaching 70% would be a genuine signal about the node rather than noise about the image cache. The comment still described the 60 GB VM /var and ended with "BARE-METAL MIGRATION: revisit this", which is now done. Set to 30 rather than deleted, deliberately. `ceph config` keeps a value that only disappears from this file: `ceph config dump` currently shows `mon advanced mon_data_avail_warn 20`, and removing the line would leave it there indefinitely. That is the trap the bluestore settings below already document, and the one that crash-looped an OSD during the network provider change when public_network survived being deleted here. mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in monitoring-rules still watches /var directly for the real risk, which is the node filling rather than the mon growing. kubectl diff against the live cluster shows the value and the generation bump, nothing else. Verify after it applies with `ceph config get mon mon_data_avail_warn` — 30 there is the proof the database entry was overwritten and not merely edited in git. Supersedes the remaining half of #503, which was closed as overtaken by the bare-metal migration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'fix: [alert] TargetDown (default) (#635 )' (#644 ) from alert-fix/issue-635 into master
2026-08-27 08:37:46 +02:00 · 2026-08-27 06:35:54 +00:00 · 2026-08-27 06:35:12 +00:00 · 2026-08-27 06:34:51 +00:00 · 2026-08-27 08:03:46 +02:00 · 2026-08-27 07:51:51 +02:00
51 changed files with 1577 additions and 1151 deletions
@@ -170,8 +170,8 @@ role keeps logs off local flash and monitors wear:
  WAL/positions sit on `/run/alloy` (tmpfs), so the agent writes nothing to
  the eMMC either. Net effect: no log writes hit local flash — the trade-off
  is that logs are RAM-only and lost on reboot unless already shipped.
- **Loki endpoint:** the unauthenticated srv-network Loki
-  (`loki.srv.renner.uno:3100`) is firewalled off the IoT VLAN, so Alloy pushes
+- **Loki endpoint:** the old srv-network Loki
+  (`loki.srv.renner.uno:3100`) no longer exists — Loki moved into the cluster, so Alloy pushes
  to the authenticated cluster ingress `https://loki.renner.uno/loki/api/v1/push`
  with basic-auth (`edge_loki_username` / `vault_edge_loki_password`).
  **Delivery only works once the `edge-1` htpasswd line has reconciled into
@@ -0,0 +1,13 @@
+$ANSIBLE_VAULT;1.1;AES256
+32373134633930343261336362373335623735656634313034393731393661643361346634623439
+3561363963333533313264643036663731656465343162330a333964383462373934303766323533
+63613065313063626261383236396363656565623538666565646661353766643133306334396561
+3961376234623236620a616339326238366538333666316538396465666365373834313064653963
+38623233616633383435373836613438333763343932636239386263306238376137643531373161
+35386437313465643437343832623935353238316565366537646530323233363837656631383962
+32653332333162623966366163633732356637323461613966313263313066643966373335313162
+30306362346561633566316166313164323635356363643034663933363464663562343136646137
+66643536363734316533663032643637356136363237376365613330633961616665323430633135
+36323733633464306330666232663433616533373466633939353061663136666665323432373666
+36393930333435333238306238373931393232393861333334643139306138653336633534626362
+61323038303838396438
@@ -1,201 +1,29 @@
 $ANSIBLE_VAULT;1.1;AES256
-32666334326535346466653862343263373365346261313136653366366664633436356536383437
-6666326436366433373163353262383031333264633530390a346539316165633939346663623264
-30653839376635626261323234326539313235623962306639396633343533363561376363363864
-3165323130633061350a346233396530303737316361653337643264383230613635383932636133
-61336561663264393831313364366665636166306437383835303332643934393062623163343861
-31643063373432643531663538666363613331626564376539336435343161333237333036343664
-31636635316631376563326637303337386337313330323838666562303463363361346531613363
-64383062396132373536663832356266383430316465643934636537396466366335636131623339
-34316166323730316463666465613734383339636264306431623535656362633739386362393134
-30343837363361333666616332646131663738663737663434303437336130313133306236313739
-36363731386163353136376230393262653334366239653032613034323333386433666262353364
-30326364343735643739626563393661616238313265633566356639623639383138303934653737
-64356139656163666263346539663733356239386333313763653434343163353436323937663034
-35653236616135383465656536323738346662396366653435383337383233643232656435356335
-63616138666662386337323865356439303961396138613965303465373766343939333933316664
-35356561353366663361306336303339633362623332386330383533336461306139613232306363
-62613731613835336338346236373636343463383433613938393865666165653935353830633966
-61393266623961343665343136323235393764313461306138343334366631313531396663666136
-34636436666238376138633461636438356461643461356139313437393736666338623038353933
-34633032653930623065623639646231346133343231383032663438326164313136613361356232
-38306565353737623532653231303963616336323563663137316139303136623234663432373661
-37333431663233656635373433323461663164303137633831306232626465303761633934336461
-65613534633665663565366634623362303964313862333936383032363137306336636563383435
-35366666313065393463656430303436346334313930616166383861613830373962353536303866
-39636165393861363161633734343838346338303036306639316231313738383831626639303833
-62316336623135336462633439323533383931666563643839383236636134363466383561633363
-64366466336362376234336463383639323165306631633437333639333263373837666565306163
-34666162336166623831353436363831666561376562613732333938353638333134323133346435
-61663261373433333635643761373465653031303236613465643931356433656161373463316331
-34326163633037633737393338313931353130306134663839623332666365373663643330353864
-36366535313032333262373038333161336230303165646133303261333839376432393435613434
-39633830316236653962663464363637333433393665353066653230383339666130653135343332
-32623165386535323832626536316665396564623064633564663437613930313665616163326637
-61363538333761633264396430326539643833313830346565643937656137376464646664313636
-38626165306661353631393031383063646463363031633435373735653838636634316634323738
-38313832376265616631623761393666346133636135666664643661646130306237383134383934
-39643561363962323261343236393630656435653261633933656262623737613130353531386438
-39643335306436636238393734646238653465626232643161316539346435353130633138393863
-63343965346361353036646561636339303063393033633162666639303166303265333233386137
-38643835616131303263366539646130643630366139623032366532653031303336323463326263
-37313566633032313931313565616131376635643030373833323838393633653531653664333032
-33623566333731636462366133653837636238383832353362653230393164343135663062373438
-61613832336633343038373565316431643432626532636435653438383536336166383436393035
-66613037663739616337383136663034643433333933643862396330356533636439666536353338
-30313761633961383337353030333862396164376337393338303565663963633662633134653362
-30346563643537323431313137636364623231376364386364633739313438376663343339333330
-64623663303535393262376437626637316532643139353262393231376135663161663862646666
-65303463393966393633613964353132303863373565353131396435623634336364313864373465
-61376335313830383534643431383966616565346466633633613933643231303732373938313434
-38383731653938656331646239353330666266336566633232353835653239393232366634663165
-31313766643539373133396563633637336133633034633236393336663064626630376632336264
-63346136613165613063316264646561633430613630396263333562366430326432323264383233
-30616631616535666631303838326330363339343463303831376531396164333539386263343334
-64313363626363316266343735313263613038383464396564313336373831393936336237303361
-65346161323961336631366263353161616562633533323036383366613637333832323466326535
-36376661346133383461626461646237373531333532656239333566323433333139633966343465
-36653130646636336534353331623062313863666637306462343139333031313234633636613232
-64383663626566616465623938346531623362383839316361366566336166386461336634303339
-63363931303866376462303634633033386332373564393932616638323764383630643839393630
-38373533636561613131343138383765376332336434303930326630303735313462316666333861
-37346336613339643466636665326466386664376438646666363537306533303036613135303230
-65303733366163363964613135306538393234636262326639613439363063326237306337666564
-32303161373661633231333662303565383134333737343530616631363337396333346261646636
-38663439653435623236366437643334393432383636316664306530303133396561396435313936
-38316637393365343936636438666461326436353636323537343363623337656536613662393333
-32346638636532333266323036623932316265303134633265366634623336386138653165643662
-35396637636136386366663762346236343565623133623336363530653039363166383534313632
-39333162396534376335646664383362366663373531646132643462396131333932653464326532
-39326633616335633030373463303665383261323763303863383566386536646136613362323034
-35313064313533616262666439663266353136616466643562666131366636623166333030346333
-62353539396131666434356238646439326337343239366234383866613065636232613863323931
-38363339623634643433393135396264363835386135303061623964343832323936313536623634
-34396531643839623932663565656535396337353934373462336231356133303335643931323131
-33383030653362323830376435393134666339333036336333376539363033666666396534636131
-64353663663633313966336439643334616332616536336238613162363435323938313636303930
-66306135613735653437366164336237656331346365646265656362316639656162386239326632
-34613431653433306633666632363639306266303361383230316539306161396566346137363133
-30333131666530636264656263303639636161366134633964393834663436326539626264383034
-34636134663833333764613237663233346363643566353166316230396235373261393739316237
-38373466623635333432326531613637303839633033636166633364383964626134386261636536
-64616432343936363664383532323936386136393332653131343533653633323639356234353363
-62613631333864663435616633396537373035646238653565646535306163623262396136356663
-63383537653231633766613231316666303830336334323738626138366434323164643465663034
-64343965363738626638363733646136316338633161383330306363643461643664303734663863
-38656135616165366438623237333161363234356538356466646266316136313832666161626566
-65306234663338323263376437363961353033396532643132336261336662656136656530343532
-61353736626665386437376230633336656663356361626437393932633765336439613665666632
-34363763656164326565633433643836383234363962613535353230356461613133666232396336
-32316264353535353162373833666365633266376634366362646264616161363634363138303832
-30656465326562653430373962346438646138336364363833646165353636346161336336613565
-37303931393865356136333431656264666331313165373037656535333832643662343434396337
-37656263363161396438626437643839633834643265313963653337363463373530333930303163
-34353163333461353763643631303763623034333665373235656531393561363432633433393137
-63333438646565633533353466373930313831316132306662363831623732336435346334323331
-32373736396239613839396535316166386637613435643637323932303239643339613737653761
-63323062333430636161316437383035326531346466336263353761636661326363656631613063
-62623863616533333036643563333463366432386232386436356632323863663862613466613931
-35636538663263336333613664353934623338356262643832356163326161373166633833333838
-36373537363934393137663462633833373364356230356136646639633737636130396231633730
-61356536373064356665646335633435646164623164333964323135343536663162656564653961
-63626432653439373938333039386661363330373136633831656230336237376436313664653633
-61623830326330663235376534313135303566636131373835383534363730616433363536346666
-38326532376338393462363430643334343137613363626132356133363534643336663063623136
-66623261343932373662663134323538666334623764376131623530373065316338653535313165
-35313337303563373737313963303134303435643563616561323963613565646466353162353637
-62623961626630323432323133666636356531316362633331313630393761303030363736643163
-63333565616232366665356535373632626366366532633036323839393266613730373331343765
-39343132643734623162386162353438353165653537303033313439326132343036333739623832
-30316132396532626236633665333162653536633134313339306661613066343937323032376235
-64386161623639636361313865613536373338333931623438366338616338326536383232363564
-30356139373865313432366337646366613361393262333336643535656438383562343763636236
-38616263613834613030363033613839613334323964343436326136626565643838323064313037
-34663964646636363038393134343634663637663030636339613864316165666265383632373637
-32373063616130343762393832663030636334646263626236663730336135376461643134646337
-63346534353264663031336164623330633737333766633662613863343431333138363337326536
-38363937313632633537363261633133636266376534346430643230623663333464393262396564
-37323632653036653336623636646537326636383865383338366138653463336638363965356536
-30373664313863396237613363373236353966303862376561653562666231353930653233383733
-38303261343436393261653436626136353237666633613166623035636466346539363436616364
-36386438666165316137303932346435313736616434643337303038383166386135326132656265
-64633866363532333465623135323761393837623934316561633637613433316334376138343764
-64646262356631373865393363353465643733623535626434666639613965383239343736356231
-31306135666539646234333934326664353137643261613131646364363739333035396437316362
-62343531656234376438376266373932376233643630383836323032633537383961383537376631
-38303062623032636264666534313266363835636132646664366162303430393161313964616633
-32613063393965653438643766373434613237393736643635653465366533363535623932386433
-38653530316364323033383330373433663865393837653466376666633434343062376432633138
-64653365373062643431353533363764666339663334393432353466623062646632383163656365
-63633365326565353730333164303237383766386135376630613234333834626335643738376339
-31626161346562613439396263646361343464636136636663363766356263636630323366326131
-37356166343031376635346433363335373137303933343539303832613363656334663964333462
-31613531383166666261336439356434323263306364616536636333316131373930646262386366
-33333032643464313165336265326130333637323461643131393437326535316164356665313636
-61613237356331393463336431316539633133326334396134663035396531633538666331393132
-64393134643832376230656332633633396333313464333066656461366335343131613038656535
-32353037316132393936373834633535646631653464616331633830326131333361316464393762
-34306136306633323766353235353663666539663036383636393335336263353134303634626131
-31316334663733666530653330333263653135643539353439306165353339623032643633393632
-37653636663164646438626638643630393135623934646166666635316532366264623165616131
-64653736306431623639303564306431616263613966656563373861616339316466383761643439
-64323836303162613931313534393436353464333330353635663034396631636563616464653530
-34616164323362653231343433323338633930323035396630326461666461346338363039383737
-38666431373633376536323739316132316231353738306632336537633938653535316530346634
-65323937336635373736346438323133353333343235623463663264623838316239366165306535
-30323136626561636537393066383539303233366262396538383563353730356537623266623038
-33316135303833323663363733353634653763383166633663313465303838646266303264363339
-30363637396161656332323333653931336661393566663232326466646563356436306332333532
-35656331383661376538633664346537613161303135643937643161383034643532653961346533
-31396237653234363832383437613433326165636262333236393334613437386638636535616634
-34313330643139386232396332376135376332613333633938366533616435303232376366623237
-37323535383932623631363431383165323338306338633066373937323361383336616265303234
-32343132303366343866303837326566383865393764343130616366373536623766316238363430
-31633734633438313065336431316161636465333166653765653962326636663065336532393731
-33386139383466626565663666373664313030663132313233356136613435663265396332656633
-33636436646565643838643930343938326231623738393661393738343062303661353362646465
-31653335343638613238333636326337396438373263663666373263616662653634613036663364
-66323639336233623538663061366462663766383539333432336363613938346263636436336430
-36646639353163316661356636393034653934666563356335396631323166313532353361653630
-61353261656431613135313965373061313038653237626231366332306432633538656462323739
-38313831663861663738336564333731363937653663663031343261633538643535336264306230
-31656365326261633730613765393861376662656137623832363832633266346135643133633331
-39623836633337323934653464333731626361316638333565393435656132626631613739366535
-37653266333432386334333461653231346630646435343230636531326338396566396466376532
-32666632383430393064353861373033393538366266333339373436353932656333366264393936
-39373938346633616534643734313934643438333965323133313037373737303964643337653135
-31623038356263643438343635643530663362656630393164663634366461623464353532353239
-38623662666235383139663162343836316331396266313062316639373034376337393563643739
-65313564303962353061616265393135326266386266316538616430383966306163386533376137
-36393565336435636535376639623931616435353532356438666663383961356638353333366139
-63313962643638323862353738316233613862346532306631386536323261636235343565353563
-64613465343662663034643138623865363862316131363461343636333864656132626166393836
-32393737386166343630323861376562353337663030613263613664663938376337333533613432
-33343064636365643831373135623466613738303164326562386430386339653831313536366532
-32613430313461663662313966373238643934633938633764363365386139303337643533313961
-61646131653638616365346236616330363566636261636362363635323566353562326239393530
-33323934623530306436623963663735353337306162646638383934326530333163396165366139
-61653339363166376266643733653536636639653939346666663834376361373533363231636433
-35363665373031393231363161333036303239313032326134363432363034623762366637643830
-32353132353961343830303562613964376234303461623263343735336238353565353238363438
-61326164323265386431353130346364393466353334386336376139303036646634376331313561
-39386331366337646634366635623066313432623537323164646263323532303465343735326364
-33383162626139383232636362613831303832633562636136393138316334303832383666313963
-37643835663065613236303338353435663830343637326563343665616532633830623035336534
-39376536663937653939326261346263613265636363333531323931646633313837386238333435
-33646339626462386635316230636135626330643630633664636364313539626332643331356334
-32313736343031633631393662363638626137303331383263656335373966383335653133333932
-34393861303665613633366461666664376466383437303065653938323166633562643834356139
-32616564373731313134316661343839333830663338636132323265376133656337393632353931
-35633531383438656461613135323134623134393062356661326435323065616331656638306432
-62313130643763626336383762306535656238393961653537336465373064653432653665373362
-63666366343866613235376535313835623261313531653265373636383030613930353530346265
-64303531316239386630626565323966616535373936393766663063623464663864363338343466
-33666237613532326661626461343166323162366532386638333835663838343865346266646533
-66343533326562396331363137353631633965623561333963613565636264653031616339613232
-39666362643739663837326432623462646233306239326463616232383236346236383132396637
-34623332346533366362613465363263373339643033663264363332336463343337333466393866
-33333363313331376634363431366636376466336131363330323530306162666434393764393563
-35633530353839386362326262343635383434333338626563383463653263333430383465626535
-63363933343365326665343166633939313739316231383938376166646331623663343161623865
-36646630336337373865
+30353730633631376539373334376531353263396339313464373538396231616166653738383161
+6438656132616365646463653464396236313437323537620a363632643061333036353537626461
+34393065613836343536383565303066326133363937386263653961663666643430663139653961
+6462363235343063350a636165376232366636363233376637636231363839343738346134623562
+63663839326435303964313730336436396365343163343366613966323938656263303166326435
+65303931393233623064343763633161353161643635396365316362333137353732346366643162
+36386132386638623333623839373161386330353635346331393735366662393935353264646439
+36386539393432373765396562613033656339646532323864366230393638313161383363636636
+32353266376563393239393734316461313566386233333164623135303132346164353165323338
+62306333363838393236346435303339393639623231353232643966613163313134336265633636
+64366539326639303362393933303465376532626663346431613563353032383037363532646637
+38376266323537343437643635343634636164393230313632663533666434306134366261383230
+66383135303565383763646331383338646339613539623835366438393635363832366134316531
+63636466653239383231616139663231623636376564376234383762383066613030313937616533
+33396135616236643034303236353230376434313365326633303961303232363766633161353434
+64333832623535343733353462313862343934633363616237353331386264386632643538333939
+34636437393934326131363236346636663665613537343134313636623439356665636663613938
+37346634356164343537396363663730306232623964636635333333653738383538343236383535
+66326462373964333234343435336630643930333433393034303439366631386666393330343238
+61336237373236313533333732343064336637326636663433396362346631343165656163313561
+30383331326231383539343639343130323731333432306630353835343338613861376631646264
+66383539616663376237663234313431313064663566313566366664386439316161353135306338
+35373962636537653535333935306564303235356232313463386237353737313736386463396164
+62666165336539316565623634326562356434313833643833353635303361353937666663356263
+35653665323736386263333437666432623332333936373938376637633463653930326466383139
+38336239646135316565356564373731386662383234386433633632393165343331353835623437
+30663138633430356334353832633434633939373337663437326263336537356633313763353337
+64373764666430643433
@@ -11,12 +11,6 @@
  ignore_unreachable: true
  gather_facts: true

-  vars:
-    # Monitoring agent endpoints
-    loki_endpoint: "http://loki.srv.renner.uno:3100"
-    prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
-
  roles:
    - common
    - qemu-guest-agent
-    - monitoring-agent
@@ -1,44 +0,0 @@
---
-# Configure Audiobookshelf permissions
-# Fixes upload issues by ensuring audiobookshelf user can write to storage directories
-
- name: Configure Audiobookshelf
-  hosts: audiobookshelf.srv.renner.uno
-  become: true
-
-  tasks:
-    - name: Ensure audiobookshelf user exists
-      ansible.builtin.user:
-        name: audiobookshelf
-        system: true
-        create_home: false
-        state: present
-
-    - name: Set ownership on audiobooks directory
-      ansible.builtin.file:
-        path: /audiobooks
-        owner: audiobookshelf
-        group: audiobookshelf
-        mode: "0755"
-        state: directory
-        recurse: false
-
-    - name: Set ownership on audiobooks content recursively
-      ansible.builtin.command:
-        cmd: chown -R audiobookshelf:audiobookshelf /audiobooks
-      changed_when: true
-
-    - name: Set ownership on metadata directory
-      ansible.builtin.file:
-        path: /metadata
-        owner: audiobookshelf
-        group: audiobookshelf
-        mode: "0755"
-        state: directory
-        recurse: true
-
-    - name: Restart audiobookshelf service
-      ansible.builtin.systemd:
-        name: audiobookshelf
-        state: restarted
-        enabled: true
@@ -10,7 +10,7 @@
 - name: Deploy the Claude Code workstation
  hosts: claude_workstation
  become: true
-  # qemu-guest-agent + monitoring-agent (Alloy -> Loki/Prometheus) are applied
+  # qemu-guest-agent is applied
  # to every inventory host by the umbrella plays/common.yml, same as for the
  # netbird-proxy host — run that against this host too (see RUNBOOK Step 5).
  roles:
@@ -18,9 +18,10 @@
 #
 # Self-contained: `certificates` and `common` are the parts of plays/common.yml
 # that apply to an edge host, carried here so bootstrapping one is a single
-# unrestricted run over the group. Not carried over: qemu-guest-agent (bare
-# metal, not VMs) and monitoring-agent (Debian-oriented; these hosts ship logs
-# via edge-log-shipper).
+# unrestricted run over the group. Not carried over: qemu-guest-agent, since
+# these are bare metal rather than VMs. Log shipping is edge-log-shipper's job
+# and lives below — it pushes to the authenticated cluster ingress with a
+# per-host credential, which is the only Loki these hosts can reach.
 #
 # Order is dependency order: edge-flash-wear before docker, so the journal is
 # volatile before containers start logging into it.
@@ -1,38 +0,0 @@
---
-# Deploy monitoring agent to NetBird server (external datacenter)
-# Uses NetBird mesh IPs to reach Loki and Prometheus
-# Usage: ansible-playbook -i inventory/ plays/setup-netbird-monitoring.yml
-
- name: Deploy Monitoring Agent to NetBird Server
-  hosts: netbird_server
-  become: yes
-  gather_facts: yes
-
-  vars:
-    # Use ARM64 binary for Hetzner ARM server
-    alloy_arch: "linux-arm64"
-
-    # Monitoring endpoints via NetBird mesh
-    loki_endpoint: "http://loki.nb.renner.uno:3100"
-    prometheus_remote_write_endpoint: "http://prometheus.nb.renner.uno:9090/api/v1/write"
-
-    # NetBird server labels
-    alloy_extra_labels:
-      environment: "production"
-      datacenter: "hetzner"
-      type: "netbird-server"
-      service: "netbird"
-
-  roles:
-    - monitoring-agent
-
-  post_tasks:
-    - name: Verify Alloy agent is running
-      systemd:
-        name: alloy
-        state: started
-      register: alloy_status
-
-    - name: Display agent status
-      debug:
-        msg: "Monitoring agent deployed to {{ inventory_hostname }} via NetBird mesh"
@@ -1,62 +0,0 @@
---
-# Setup PostgreSQL monitoring user and exporter
-# Usage: ansible-playbook -i inventory/ plays/setup-postgres-monitoring.yml
-
- name: Create PostgreSQL monitoring user on primary
-  hosts: authentik-db-1.srv.renner.uno
-  become: yes
-  vars_files:
-    - ../inventory/group_vars/all/vault.yml
-  tasks:
-    - name: Install psycopg2 for Ansible PostgreSQL modules
-      apt:
-        name: python3-psycopg2
-        state: present
-
-    - name: Create postgres_exporter user
-      become_user: postgres
-      community.postgresql.postgresql_user:
-        name: postgres_exporter
-        password: "{{ vault_postgres_exporter_password }}"
-        role_attr_flags: "NOSUPERUSER,NOCREATEDB,NOCREATEROLE"
-        state: present
-
-    - name: Grant pg_monitor role to exporter user
-      become_user: postgres
-      community.postgresql.postgresql_privs:
-        database: postgres
-        type: group
-        objs: pg_monitor
-        roles: postgres_exporter
-        grant_option: no
-
-    - name: Grant connect on authentik database
-      become_user: postgres
-      community.postgresql.postgresql_privs:
-        database: authentik
-        type: database
-        privs: CONNECT
-        roles: postgres_exporter
-
- name: Deploy postgres-exporter to Prometheus server
-  hosts: prometheus.srv.renner.uno
-  become: yes
-  vars_files:
-    - ../inventory/group_vars/all/vault.yml
-  vars:
-    postgres_exporter_db_host: "10.2.40.135"  # PostgreSQL HA VIP
-    postgres_exporter_db_user: "postgres_exporter"
-    postgres_exporter_db_password: "{{ vault_postgres_exporter_password }}"
-    postgres_exporter_db_name: "postgres"
-  roles:
-    - postgres-exporter
-
-  post_tasks:
-    - name: Verify postgres-exporter is running
-      uri:
-        url: "http://127.0.0.1:9187/metrics"
-        status_code: 200
-      register: exporter_health
-      retries: 3
-      delay: 5
-      until: exporter_health.status == 200
@@ -0,0 +1,250 @@
+---
+# Rotate the NetBird shared secrets that Ansible owns BOTH ends of.
+#
+# Deliberately opt-in. Rotation is not idempotent and it interrupts the mesh
+# briefly, so this play refuses to run unless invoked explicitly:
+#
+#   ansible-playbook plays/rotate-netbird-secrets.yml -e rotate_confirm=yes
+#
+# ONE-TIME MIGRATION REQUIRED BEFORE FIRST USE. The three keys below currently
+# live in group_vars/all/vault.yml. They must be deleted from there, because
+# group_vars files merge alphabetically and vault.yml sorts after
+# netbird-rotatable.yml — leaving them would let a later ordinary deploy
+# overwrite a freshly rotated secret with the old one. The play checks for this
+# and refuses to run until it is done:
+#
+#   ansible-vault edit inventory/group_vars/all/vault.yml
+#   # delete vault_netbird_turn_secret, vault_netbird_relay_secret,
+#   #        and netbird_turn_secret
+#
+# That last one is easy to miss and breaks the rotation silently. vault.yml
+# carried BOTH vault_netbird_turn_secret and a bare netbird_turn_secret. The
+# role default is `netbird_turn_secret: "{{ vault_netbird_turn_secret }}"`, but
+# a group var outranks a role default — so the bare key wins, and rotating only
+# the vault_-prefixed one changes nothing that reaches the template. The check
+# below matches both spellings for that reason.
+#
+# The first rotation then populates netbird-rotatable.yml with fresh values, so
+# nothing needs copying across.
+#
+# WHAT IS ROTATED, and why each is safe to rotate from here:
+#
+#   vault_netbird_turn_secret   coturn's `user=netbird:<secret>` in
+#                               turnserver.conf and the management server's
+#                               Turn.Secret/Password in management.json. Both
+#                               templates, one role, one host — the two ends
+#                               cannot disagree across a run.
+#   vault_netbird_relay_secret  management.json on the server and relay.env on
+#                               the proxies. Two hosts, both managed here.
+#
+# The test for "safe to rotate here" is whether the value appears in
+# management.json.j2. If the management server does not read it from a file
+# Ansible writes, Ansible does not own both ends, whatever else it templates.
+#
+# WHAT IS NOT, and must not be added:
+#
+#   vault_netbird_proxy_token    LOOKS symmetric — it is in proxy.env on both
+#                                the server and the proxies, all Ansible
+#                                templates — but the management server does not
+#                                validate it from any of them. It is registered
+#                                in the management DATABASE. Rotating it changes
+#                                only what the proxies present, so every proxy
+#                                is locked out: "proxy auth failed: too many
+#                                failed authentication attempts", and the
+#                                proxies stop receiving their domain
+#                                configuration. Learned by doing it.
+#
+#   vault_netbird_datastore_key  encrypts NetBird's datastore at rest. Changing
+#                                it without re-encrypting the store makes the
+#                                existing data unreadable. Rotating this needs a
+#                                dump/restore, not a template rewrite.
+#   vault_netbird_auth_client_secret  the other half lives in Authentik. Ansible
+#                                only writes the consumer side, so rotating here
+#                                breaks login until Authentik is changed too.
+#
+# These are Ansible's OWN credentials, not a copy of the cluster's. OpenBao
+# carries credentials for some of the same systems (see
+# kubernetes/infrastructure/openbao/rotation/), and those are deliberately DIFFERENT
+# values: one consumer, one credential, independently revocable. Rotating here
+# must not touch OpenBao and does not invalidate anything the cluster holds —
+# that is the point of them being distinct, not drift to be reconciled.
+#
+# Corollary: if a credential ever has to be shared between the two, that is the
+# bug. Mint a second one instead.
+#
+# There IS a brief interruption. The server is reconfigured before the proxies,
+# so relay/proxy auth fails for the seconds between the two plays. Clients
+# reconnect; existing peer-to-peer tunnels are not torn down.
+
+- name: Generate and store new NetBird shared secrets
+  hosts: localhost
+  connection: local
+  gather_facts: false
+
+  vars:
+    rotatable_vault: "{{ playbook_dir }}/../inventory/group_vars/all/netbird-rotatable.yml"
+
+  pre_tasks:
+    - name: Require explicit confirmation
+      ansible.builtin.assert:
+        that:
+          - rotate_confirm | default('') == 'yes'
+        fail_msg: >-
+          Refusing to rotate without -e rotate_confirm=yes. This changes live
+          credentials and briefly interrupts the NetBird mesh.
+        success_msg: Confirmed; rotating NetBird shared secrets.
+
+    - name: Verify the vault password is available before touching anything
+      ansible.builtin.command:
+        cmd: "{{ playbook_dir }}/../vault-password.sh"
+      register: _vault_pw_check
+      changed_when: false
+      no_log: true
+
+    # Files inside group_vars/<group>/ are merged ALPHABETICALLY and the last
+    # one wins. netbird-rotatable.yml sorts before vault.yml, so if vault.yml
+    # still defines these keys it silently overrides everything this play
+    # writes. The rotation itself would still work — the plays below pass the
+    # new values explicitly, and play vars outrank group vars — but the NEXT
+    # ordinary run of install-netbird-server.yml or install-netbird-proxy.yml
+    # would read the stale value from vault.yml and push it back out, breaking
+    # server<->proxy auth. The deploy pipelines run unattended, so that would
+    # land on an unrelated change to those roles.
+    #
+    # Counting matches rather than printing them: the value never reaches
+    # stdout, only the key name does.
+    # Proving the vault is readable FIRST. Without this, a failed decrypt (wrong
+    # password, moved file) sends empty input to grep, which then reports zero
+    # matches — indistinguishable from a clean vault, so the guard below would
+    # wave the rotation through precisely when it knows least.
+    - name: Prove vault.yml is readable before trusting the key check
+      ansible.builtin.shell:
+        cmd: >-
+          set -o pipefail;
+          ansible-vault view
+          --vault-password-file {{ playbook_dir }}/../vault-password.sh
+          {{ playbook_dir }}/../inventory/group_vars/all/vault.yml
+          | wc -l
+        executable: /bin/bash
+      register: _vault_lines
+      changed_when: false
+
+    - name: Check vault.yml does not still define the rotated keys
+      ansible.builtin.shell:
+        cmd: >-
+          ansible-vault view
+          --vault-password-file {{ playbook_dir }}/../vault-password.sh
+          {{ playbook_dir }}/../inventory/group_vars/all/vault.yml
+          | grep -cE '^(vault_)?netbird_(turn_secret|relay_secret):'
+      register: _stale_keys
+      changed_when: false
+      # grep exits 1 on "no matches", which is the good case here.
+      failed_when: false
+
+    - name: Refuse to rotate while vault.yml still owns these keys
+      ansible.builtin.assert:
+        that:
+          - _vault_lines.stdout | int > 0
+          - _stale_keys.stdout | int == 0
+        fail_msg: >-
+          vault.yml still defines {{ _stale_keys.stdout }} of the keys this play
+          rotates. group_vars files merge alphabetically and vault.yml sorts
+          last, so it would override netbird-rotatable.yml on the next deploy
+          and revert the secrets. Remove vault_netbird_turn_secret,
+          netbird_turn_secret and vault_netbird_relay_secret from vault.yml
+          first (one-time migration), then re-run.
+        success_msg: vault.yml does not shadow the rotated keys.
+
+  tasks:
+    # 48 alphanumerics. coturn's shared secret and the relay secret are both
+    # opaque strings; length is the only property that matters.
+    #
+    # ansible.builtin.password, NOT community.general.random_string: the CI
+    # image (ci/images/ansible/Containerfile) installs only community.proxmox
+    # and community.docker, so anything from community.general fails at this
+    # task when the play runs through the project's own tooling — even though
+    # it resolves fine on a workstation that happens to have the collection.
+    #
+    # `/dev/null` as the path makes the lookup generate without persisting;
+    # the value exists only in this fact.
+    - name: Generate replacement secrets
+      ansible.builtin.set_fact:
+        new_turn_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
+        new_relay_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
+      no_log: true
+
+    - name: Render the new values to a temporary file
+      ansible.builtin.copy:
+        dest: "{{ rotatable_vault }}.new"
+        mode: "0600"
+        content: |
+          ---
+          # Generated by plays/rotate-netbird-secrets.yml. Do not hand-edit —
+          # the next rotation overwrites this file wholesale.
+          vault_netbird_turn_secret: "{{ new_turn_secret }}"
+          vault_netbird_relay_secret: "{{ new_relay_secret }}"
+      no_log: true
+
+    # Encrypting the whole file rather than inline !vault strings: these three
+    # values are always rewritten together, so per-value encryption buys nothing
+    # and makes the write non-atomic.
+    - name: Encrypt the new file
+      ansible.builtin.command:
+        cmd: >-
+          ansible-vault encrypt
+          --vault-password-file {{ playbook_dir }}/../vault-password.sh
+          {{ rotatable_vault }}.new
+      changed_when: true
+
+    # Move into place only after encryption succeeded, so a failure above never
+    # leaves plaintext where the inventory would load it.
+    - name: Install the encrypted file
+      ansible.builtin.command:
+        cmd: "mv {{ rotatable_vault }}.new {{ rotatable_vault }}"
+      changed_when: true
+
+    - name: Confirm the installed file is encrypted
+      ansible.builtin.command:
+        cmd: "head -1 {{ rotatable_vault }}"
+      register: _vault_header
+      changed_when: false
+      failed_when: "'$ANSIBLE_VAULT' not in _vault_header.stdout"
+
+- name: Apply the new secrets to the NetBird server
+  hosts: netbird_server
+  become: true
+
+  vars:
+    netbird_turn_secret: "{{ hostvars['localhost']['new_turn_secret'] }}"
+    netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
+
+  roles:
+    - netbird-server
+
+- name: Apply the new secrets to the NetBird proxies
+  hosts: netbird_proxy
+  become: true
+  # One at a time so at least one relay stays reachable while the other
+  # restarts. The server is already on the new secret at this point, so a
+  # proxy is broken only between its own template write and its restart.
+  serial: 1
+
+  vars:
+    netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
+
+  roles:
+    - netbird-proxy
+
+- name: Report what still needs doing by hand
+  hosts: localhost
+  connection: local
+  gather_facts: false
+
+  tasks:
+    - name: Remaining manual steps
+      ansible.builtin.debug:
+        msg:
+          - "Rotated: turn_secret and relay_secret."
+          - "Commit inventory/group_vars/all/netbird-rotatable.yml — it is encrypted."
+          - "NOT rotated: datastore_key (needs dump/restore), auth_client_secret (Authentik side)."
+          - "OpenBao is unaffected by design — its NetBird credentials are separate ones."
@@ -169,8 +169,7 @@ Merge the PR, then trigger the pipeline (Woodpecker → Run → `PIPELINE ==
 "deploy-claude-workstation"`), or push a change under the role's paths. CI SSHes
 to `10.3.40.60` over the tunnel and applies `common`, `netbird-client`, and this
 role. Also run the umbrella `plays/common.yml` against the new host once, so it
-picks up `qemu-guest-agent` and `monitoring-agent` (Alloy → Loki/Prometheus)
-like every other inventory host.
+picks up `qemu-guest-agent` like every other inventory host.

 ---

@@ -3,8 +3,8 @@
 # Debian 12 VM on the off-site nas-2 (Synology VMM). See README for the manual
 # bootstrap (VM creation, Netbird route, secret material) that precedes this.
 #
-# common.yml already applies `common`, `qemu-guest-agent` and `monitoring-agent`
-# (Alloy -> Loki/Prometheus) to every inventory host, so this role owns only the
+# common.yml already applies `common` and `qemu-guest-agent`
+# to every inventory host, so this role owns only the
 # Claude-specific surface: toolchain, the two credential-isolated users, the
 # read-only creds, notes sync, the unattended timers, and egress lockdown.

@@ -1,8 +1,8 @@
 ---
 # Journal -> Loki shipping.
 #
-# The srv-network Loki (loki.srv.renner.uno:3100, no auth) is firewalled off
-# the IoT VLAN; the only endpoint these hosts can reach is the authenticated
+# The srv-network Loki (loki.srv.renner.uno:3100) no longer exists — it moved
+# into the cluster. The only endpoint these hosts can reach is the authenticated
 # cluster ingress. edge_loki_password comes from the vault.

 edge_alloy_config_file: /etc/alloy/edge.alloy
@@ -7,8 +7,8 @@
 // Loki. Its own WAL/positions live on /run (tmpfs, see edge-alloy.service),
 // so this agent writes nothing to the eMMC either.
 //
-// The srv-network Loki (loki.srv.renner.uno:3100, unauthenticated) is not
-// reachable from the IoT VLAN; the only reachable endpoint is the cluster
+// The srv-network Loki (loki.srv.renner.uno:3100) no longer exists — it moved
+// into the cluster. The only reachable endpoint is the cluster
 // ingress, which enforces Traefik basic-auth — hence the credential below.

 loki.relabel "journal" {
@@ -1,59 +0,0 @@
---
-# Monitoring Agent Configuration (Grafana Alloy)
-
-# Enable/disable agent deployment
-monitoring_agent_enabled: true
-
-# Version and download
-alloy_version: "1.13.2"
-alloy_arch: "linux-amd64"
-alloy_download_url: "https://github.com/grafana/alloy/releases/download/v{{ alloy_version }}/alloy-{{ alloy_arch }}.zip"
-alloy_bin_path: "/usr/local/bin/alloy"
-
-# Directories
-alloy_config_dir: "/etc/alloy"
-alloy_config_file: "{{ alloy_config_dir }}/config.alloy"
-alloy_data_dir: "/var/lib/alloy"
-
-# Endpoints - MUST be configured in group_vars
-loki_endpoint: "http://loki.srv.renner.uno:3100"
-prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
-
-# Collection settings
-alloy_collect_journal: true
-alloy_collect_syslog: true
-alloy_collect_node_metrics: true
-alloy_collect_docker: "{{ docker_socket_exists | default(false) }}"
-
-# Docker container monitoring (cAdvisor)
-# Set to true on hosts that should have cAdvisor deployed
-docker_monitoring_enabled: false
-
-# Scrape interval
-alloy_scrape_interval: "30s"
-
-# Labels applied to all metrics/logs
-alloy_extra_labels:
-  environment: "production"
-  datacenter: "homelab"
-
-# Systemd service
-alloy_user: "alloy"
-alloy_group: "alloy"
-
-# Agent port for self-metrics
-alloy_http_listen_port: 12345
-
-# Extra scrape targets for custom exporters
-# Format: list of {name: "...", address: "host:port", job: "..."}
-alloy_extra_scrape_targets: []
-
-# Syslog receiver (for receiving logs from remote devices like Synology NAS)
-# Enable this on the central log collection host (e.g., loki.srv.renner.uno)
-alloy_syslog_receiver_enabled: false
-alloy_syslog_receiver_port: 1514  # Use non-privileged port, Synology can specify port
-alloy_syslog_receiver_protocol: "udp"  # udp or tcp
-alloy_syslog_receiver_format: "rfc5424"  # rfc3164, rfc5424, or raw (requires experimental)
-
-# Stability level for experimental features (e.g., raw syslog format)
-alloy_stability_level: "generally-available"  # generally-available, public-preview, experimental
@@ -1,17 +0,0 @@
---
- name: Restart alloy
-  systemd:
-    name: alloy
-    state: restarted
-    daemon_reload: yes
-  when: monitoring_agent_enabled | bool
-
- name: Reload systemd
-  systemd:
-    daemon_reload: yes
-
- name: Restart cadvisor
-  systemd:
-    name: cadvisor
-    state: restarted
-    daemon_reload: yes
@@ -1,170 +0,0 @@
---
-# Monitoring Agent (Grafana Alloy) Installation and Configuration
-# This role is designed to be included in common.yml for all hosts
-
- name: Skip if monitoring agent is disabled
-  meta: end_host
-  when: not (monitoring_agent_enabled | bool)
-
- name: Install dependencies
-  apt:
-    name:
-      - unzip
-      - curl
-    state: present
-    update_cache: yes
-  when: ansible_facts['os_family'] == 'Debian'
-
- name: Check if Docker socket exists
-  stat:
-    path: /var/run/docker.sock
-  register: docker_socket
-
- name: Set Docker collection fact
-  set_fact:
-    alloy_collect_docker: "{{ docker_socket.stat.exists }}"
-
- name: Deploy cAdvisor container for Docker metrics
-  when: docker_monitoring_enabled | bool and docker_socket.stat.exists
-  block:
-    - name: Create cAdvisor systemd service
-      copy:
-        dest: /etc/systemd/system/cadvisor.service
-        mode: '0644'
-        content: |
-          [Unit]
-          Description=cAdvisor Container Monitoring
-          After=docker.service
-          Requires=docker.service
-
-          [Service]
-          Type=simple
-          Restart=always
-          RestartSec=10
-          ExecStartPre=-/usr/bin/docker rm -f cadvisor
-          ExecStart=/usr/bin/docker run --rm \
-            --name=cadvisor \
-            --privileged \
-            --device=/dev/kmsg \
-            --volume=/:/rootfs:ro \
-            --volume=/var/run:/var/run:ro \
-            --volume=/sys:/sys:ro \
-            --volume=/var/lib/docker/:/var/lib/docker:ro \
-            --volume=/dev/disk/:/dev/disk:ro \
-            --volume=/cgroup:/cgroup:ro \
-            --publish=127.0.0.1:8080:8080 \
-            gcr.io/cadvisor/cadvisor:v0.49.1 \
-            --store_container_labels=true
-          ExecStop=/usr/bin/docker stop cadvisor
-
-          [Install]
-          WantedBy=multi-user.target
-      notify:
-        - Reload systemd
-        - Restart cadvisor
-
-    - name: Enable and start cAdvisor service
-      systemd:
-        name: cadvisor
-        state: started
-        enabled: yes
-        daemon_reload: yes
-
- name: Create alloy user
-  user:
-    name: "{{ alloy_user }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    create_home: no
-    groups: "{{ 'docker' if alloy_collect_docker else omit }}"
-    append: yes
-
- name: Create directories
-  file:
-    path: "{{ item }}"
-    state: directory
-    owner: "{{ alloy_user }}"
-    group: "{{ alloy_group }}"
-    mode: '0755'
-  loop:
-    - "{{ alloy_config_dir }}"
-    - "{{ alloy_data_dir }}"
-
- name: Check if Alloy is already installed
-  stat:
-    path: "{{ alloy_bin_path }}"
-  register: alloy_binary
-
- name: Get installed Alloy version
-  command: "{{ alloy_bin_path }} --version"
-  register: installed_version
-  changed_when: false
-  failed_when: false
-  when: alloy_binary.stat.exists
-
- name: Download Alloy
-  get_url:
-    url: "{{ alloy_download_url }}"
-    dest: "/tmp/alloy.zip"
-    mode: '0644'
-  when: not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
-
- name: Unzip Alloy
-  unarchive:
-    src: "/tmp/alloy.zip"
-    dest: "/tmp"
-    remote_src: yes
-  when: not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
-
- name: Find Alloy binary in extracted files
-  find:
-    paths: "/tmp"
-    patterns: "alloy-{{ alloy_arch }}"
-    file_type: file
-  register: alloy_extracted_binary
-  when: not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
-
- name: Install Alloy binary from extracted file
-  copy:
-    src: "{{ alloy_extracted_binary.files[0].path }}"
-    dest: "{{ alloy_bin_path }}"
-    mode: '0755'
-    remote_src: yes
-  when:
-    - not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
-    - alloy_extracted_binary.files is defined and alloy_extracted_binary.files | length > 0
-  notify: Restart alloy
-
- name: Create Alloy configuration
-  template:
-    src: config.alloy.j2
-    dest: "{{ alloy_config_file }}"
-    owner: "{{ alloy_user }}"
-    group: "{{ alloy_group }}"
-    mode: '0644'
-  notify: Restart alloy
-
- name: Create Alloy systemd service
-  template:
-    src: alloy.service.j2
-    dest: /etc/systemd/system/alloy.service
-    mode: '0644'
-  notify:
-    - Reload systemd
-    - Restart alloy
-
- name: Enable and start Alloy service
-  systemd:
-    name: alloy
-    state: started
-    enabled: yes
-    daemon_reload: yes
-
- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - "/tmp/alloy.zip"
-    - "/tmp/alloy-{{ alloy_arch }}"
-  ignore_errors: yes
@@ -1,37 +0,0 @@
-[Unit]
-Description=Grafana Alloy Monitoring Agent
-Documentation=https://grafana.com/docs/alloy/latest/
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ alloy_user }}
-Group={{ alloy_group }}
-ExecStart={{ alloy_bin_path }} run \
-  --server.http.listen-addr=0.0.0.0:{{ alloy_http_listen_port }} \
-  --storage.path={{ alloy_data_dir }} \
-{% if alloy_stability_level is defined and alloy_stability_level != "generally-available" %}
-  --stability.level={{ alloy_stability_level }} \
-{% endif %}
-  {{ alloy_config_file }}
-ExecReload=/bin/kill -HUP $MAINPID
-Restart=on-failure
-RestartSec=10s
-LimitNOFILE=65536
-
-{% if ansible_virtualization_type is defined and ansible_virtualization_type != 'lxc' %}
-# Security hardening (disabled in LXC containers due to namespace restrictions)
-NoNewPrivileges=yes
-ProtectSystem=strict
-ProtectHome=yes
-ReadWritePaths={{ alloy_data_dir }} /var/log
-ReadOnlyPaths=/var/run/docker.sock
-PrivateTmp=yes
-{% endif %}
-
-# Allow reading journal
-SupplementaryGroups=systemd-journal{% if alloy_collect_docker %} docker{% endif %}
-
-[Install]
-WantedBy=multi-user.target
@@ -1,266 +0,0 @@
-// Grafana Alloy Configuration
-// Host: {{ inventory_hostname }}
-// Generated by Ansible
-
-// =====================================================
-// Logging Configuration (Loki)
-// =====================================================
-
-{% if alloy_collect_journal %}
-// Collect systemd journal logs
-loki.source.journal "journal" {
-  forward_to = [loki.write.default.receiver]
-  labels = {
-    job = "systemd-journal",
-    host = "{{ inventory_hostname }}",
-{% for key, value in alloy_extra_labels.items() %}
-    {{ key }} = "{{ value }}",
-{% endfor %}
-  }
-  relabel_rules = loki.relabel.journal.rules
-}
-
-loki.relabel "journal" {
-  forward_to = []
-
-  rule {
-    source_labels = ["__journal__systemd_unit"]
-    target_label  = "unit"
-  }
-
-  rule {
-    source_labels = ["__journal_priority_keyword"]
-    target_label  = "level"
-  }
-
-  rule {
-    source_labels = ["__journal__hostname"]
-    target_label  = "hostname"
-  }
-}
-{% endif %}
-
-{% if alloy_collect_syslog %}
-// Collect syslog files
-local.file_match "syslog" {
-  path_targets = [
-    {__path__ = "/var/log/syslog", job = "syslog"},
-    {__path__ = "/var/log/auth.log", job = "auth"},
-    {__path__ = "/var/log/kern.log", job = "kernel"},
-  ]
-}
-
-loki.source.file "syslog" {
-  targets    = local.file_match.syslog.targets
-  forward_to = [loki.write.default.receiver]
-  tail_from_end = true
-}
-{% endif %}
-
-{% if alloy_collect_docker %}
-// Docker container logs
-discovery.docker "containers" {
-  host = "unix:///var/run/docker.sock"
-}
-
-discovery.relabel "docker" {
-  targets = discovery.docker.containers.targets
-
-  rule {
-    source_labels = ["__meta_docker_container_name"]
-    target_label  = "container"
-    regex         = "/(.*)"
-  }
-
-  rule {
-    source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
-    target_label  = "service"
-  }
-
-  rule {
-    source_labels = ["__meta_docker_container_label_com_docker_compose_project"]
-    target_label  = "project"
-  }
-}
-
-loki.source.docker "docker" {
-  host       = "unix:///var/run/docker.sock"
-  targets    = discovery.relabel.docker.output
-  forward_to = [loki.process.docker.receiver]
-  labels = {
-    job  = "docker",
-    host = "{{ inventory_hostname }}",
-{% for key, value in alloy_extra_labels.items() %}
-    {{ key }} = "{{ value }}",
-{% endfor %}
-  }
-}
-
-loki.process "docker" {
-  forward_to = [loki.write.default.receiver]
-
-  stage.json {
-    expressions = {
-      log    = "log",
-      stream = "stream",
-      time   = "time",
-    }
-  }
-
-  stage.labels {
-    values = {
-      stream = "",
-    }
-  }
-
-  stage.output {
-    source = "log"
-  }
-}
-{% endif %}
-
-{% if alloy_syslog_receiver_enabled %}
-// Syslog receiver for remote devices (Synology NAS, network devices, etc.)
-loki.source.syslog "syslog_receiver" {
-  listener {
-    address       = "0.0.0.0:{{ alloy_syslog_receiver_port }}"
-    protocol      = "{{ alloy_syslog_receiver_protocol }}"
-    syslog_format = "{{ alloy_syslog_receiver_format }}"
-    labels        = {
-      job = "syslog-remote",
-    }
-  }
-{% if alloy_syslog_receiver_format == "raw" %}
-  forward_to = [loki.process.syslog_receiver.receiver]
-}
-
-// Parse raw syslog messages (BSD/RFC 3164 format)
-// Synology's "IETF" format actually sends: <priority>MMM DD HH:MM:SS hostname app[pid]: message
-// Example: <14>Mar  1 09:02:47 nas-1 Connection: User [pve] from...
-loki.process "syslog_receiver" {
-  forward_to = [loki.write.default.receiver]
-
-  stage.regex {
-    expression = "^\\w+\\s+\\d+\\s+\\d+:\\d+:\\d+\\s+(?P<hostname>\\S+)\\s+(?P<app>[^:\\[]+)(?:\\[\\d+\\])?:\\s*(?P<message>.*)$"
-  }
-
-  stage.labels {
-    values = {
-      hostname = "",
-      app      = "",
-    }
-  }
-
-  stage.output {
-    source = "message"
-  }
-}
-{% else %}
-  forward_to = [loki.write.default.receiver]
-}
-{% endif %}
-{% endif %}
-
-// Write logs to Loki
-loki.write "default" {
-  endpoint {
-    url = "{{ loki_endpoint }}/loki/api/v1/push"
-  }
-  external_labels = {
-    host = "{{ inventory_hostname }}",
-{% for key, value in alloy_extra_labels.items() %}
-    {{ key }} = "{{ value }}",
-{% endfor %}
-  }
-}
-
-// =====================================================
-// Metrics Configuration (Prometheus)
-// =====================================================
-
-{% if alloy_collect_node_metrics %}
-// Node metrics (CPU, memory, disk, network)
-prometheus.exporter.unix "node" {
-  // Using default settings for comprehensive node metrics
-}
-
-prometheus.scrape "node" {
-  targets    = prometheus.exporter.unix.node.targets
-  forward_to = [prometheus.remote_write.default.receiver]
-  scrape_interval = "{{ alloy_scrape_interval }}"
-  job_name = "node"
-}
-{% endif %}
-
-{% if alloy_collect_docker and docker_monitoring_enabled %}
-// Docker/cAdvisor metrics - scrape from cAdvisor container
-prometheus.scrape "cadvisor" {
-  targets = [
-    {"__address__" = "127.0.0.1:8080"},
-  ]
-  forward_to = [prometheus.relabel.cadvisor.receiver]
-  scrape_interval = "{{ alloy_scrape_interval }}"
-  job_name = "cadvisor"
-}
-
-// Relabel cAdvisor metrics to add host label
-prometheus.relabel "cadvisor" {
-  forward_to = [prometheus.remote_write.default.receiver]
-
-  rule {
-    target_label = "host"
-    replacement  = "{{ inventory_hostname }}"
-  }
-}
-{% endif %}
-
-// Self-monitoring metrics
-prometheus.exporter.self "alloy" {}
-
-prometheus.scrape "alloy_self" {
-  targets    = prometheus.exporter.self.alloy.targets
-  forward_to = [prometheus.remote_write.default.receiver]
-  scrape_interval = "{{ alloy_scrape_interval }}"
-  job_name = "alloy"
-}
-
-{% if alloy_extra_scrape_targets | length > 0 %}
-// Extra scrape targets (custom exporters)
-{% for target in alloy_extra_scrape_targets %}
-prometheus.scrape "{{ target.name }}" {
-  targets = [
-    {"__address__" = "{{ target.address }}"},
-  ]
-  forward_to = [prometheus.relabel.extra_{{ target.name }}.receiver]
-  scrape_interval = "{{ alloy_scrape_interval }}"
-  job_name = "{{ target.job }}"
-}
-
-prometheus.relabel "extra_{{ target.name }}" {
-  forward_to = [prometheus.remote_write.default.receiver]
-
-  rule {
-    target_label = "host"
-    replacement  = "{{ inventory_hostname }}"
-  }
-}
-{% endfor %}
-{% endif %}
-
-// Remote write to Prometheus
-prometheus.remote_write "default" {
-  endpoint {
-    url = "{{ prometheus_remote_write_endpoint }}"
-  }
-  external_labels = {
-    host = "{{ inventory_hostname }}",
-{% for key, value in alloy_extra_labels.items() %}
-    {{ key }} = "{{ value }}",
-{% endfor %}
-  }
-}
-
-// =====================================================
-// Agent HTTP Server (for health checks and metrics)
-// =====================================================
-// Agent serves metrics on port {{ alloy_http_listen_port }}
@@ -1,19 +0,0 @@
---
-# Postgres Exporter defaults
-
-postgres_exporter_version: "0.15.0"
-postgres_exporter_port: 9187
-postgres_exporter_user: postgres-exporter
-postgres_exporter_group: postgres-exporter
-
-# PostgreSQL connection (use vault for password)
-postgres_exporter_db_host: "localhost"
-postgres_exporter_db_port: 5432
-postgres_exporter_db_user: "postgres_exporter"
-postgres_exporter_db_password: "{{ vault_postgres_exporter_password | default('changeme') }}"
-postgres_exporter_db_name: "postgres"
-postgres_exporter_db_sslmode: "disable"
-
-# Installation paths
-postgres_exporter_install_dir: /opt/postgres-exporter
-postgres_exporter_binary: "{{ postgres_exporter_install_dir }}/postgres_exporter"
@@ -1,9 +0,0 @@
---
- name: Reload systemd
-  systemd:
-    daemon_reload: yes
-
- name: Restart postgres-exporter
-  systemd:
-    name: postgres-exporter
-    state: restarted
@@ -1,97 +0,0 @@
---
-# Install and configure Postgres Exporter
-
- name: Create postgres-exporter system group
-  group:
-    name: "{{ postgres_exporter_group }}"
-    system: yes
-    state: present
-
- name: Create postgres-exporter system user
-  user:
-    name: "{{ postgres_exporter_user }}"
-    group: "{{ postgres_exporter_group }}"
-    system: yes
-    shell: /usr/sbin/nologin
-    home: "{{ postgres_exporter_install_dir }}"
-    createhome: no
-
- name: Create installation directory
-  file:
-    path: "{{ postgres_exporter_install_dir }}"
-    state: directory
-    owner: "{{ postgres_exporter_user }}"
-    group: "{{ postgres_exporter_group }}"
-    mode: '0755'
-
- name: Check if postgres_exporter is already installed
-  stat:
-    path: "{{ postgres_exporter_binary }}"
-  register: postgres_exporter_installed
-
- name: Get installed version
-  command: "{{ postgres_exporter_binary }} --version"
-  register: installed_version
-  when: postgres_exporter_installed.stat.exists
-  changed_when: false
-  failed_when: false
-
- name: Download postgres_exporter
-  get_url:
-    url: "https://github.com/prometheus-community/postgres_exporter/releases/download/v{{ postgres_exporter_version }}/postgres_exporter-{{ postgres_exporter_version }}.linux-amd64.tar.gz"
-    dest: /tmp/postgres_exporter.tar.gz
-    mode: '0644'
-  when: not postgres_exporter_installed.stat.exists or (installed_version.stdout is defined and postgres_exporter_version not in installed_version.stdout)
-
- name: Extract postgres_exporter
-  unarchive:
-    src: /tmp/postgres_exporter.tar.gz
-    dest: /tmp
-    remote_src: yes
-  when: not postgres_exporter_installed.stat.exists or (installed_version.stdout is defined and postgres_exporter_version not in installed_version.stdout)
-
- name: Install postgres_exporter binary
-  copy:
-    src: "/tmp/postgres_exporter-{{ postgres_exporter_version }}.linux-amd64/postgres_exporter"
-    dest: "{{ postgres_exporter_binary }}"
-    owner: "{{ postgres_exporter_user }}"
-    group: "{{ postgres_exporter_group }}"
-    mode: '0755'
-    remote_src: yes
-  when: not postgres_exporter_installed.stat.exists or (installed_version.stdout is defined and postgres_exporter_version not in installed_version.stdout)
-  notify: Restart postgres-exporter
-
- name: Create postgres_exporter environment file
-  template:
-    src: postgres_exporter.env.j2
-    dest: "{{ postgres_exporter_install_dir }}/postgres_exporter.env"
-    owner: "{{ postgres_exporter_user }}"
-    group: "{{ postgres_exporter_group }}"
-    mode: '0600'
-  notify: Restart postgres-exporter
-
- name: Create systemd service
-  template:
-    src: postgres_exporter.service.j2
-    dest: /etc/systemd/system/postgres-exporter.service
-    owner: root
-    group: root
-    mode: '0644'
-  notify:
-    - Reload systemd
-    - Restart postgres-exporter
-
- name: Enable and start postgres-exporter service
-  systemd:
-    name: postgres-exporter
-    enabled: yes
-    state: started
-    daemon_reload: yes
-
- name: Clean up downloaded files
-  file:
-    path: "{{ item }}"
-    state: absent
-  loop:
-    - /tmp/postgres_exporter.tar.gz
-    - "/tmp/postgres_exporter-{{ postgres_exporter_version }}.linux-amd64"
@@ -1,2 +0,0 @@
-# PostgreSQL connection string for postgres_exporter
-DATA_SOURCE_NAME="postgresql://{{ postgres_exporter_db_user }}:{{ postgres_exporter_db_password }}@{{ postgres_exporter_db_host }}:{{ postgres_exporter_db_port }}/{{ postgres_exporter_db_name }}?sslmode={{ postgres_exporter_db_sslmode }}"
@@ -1,23 +0,0 @@
-[Unit]
-Description=Prometheus PostgreSQL Exporter
-Documentation=https://github.com/prometheus-community/postgres_exporter
-After=network-online.target
-Wants=network-online.target
-
-[Service]
-Type=simple
-User={{ postgres_exporter_user }}
-Group={{ postgres_exporter_group }}
-EnvironmentFile={{ postgres_exporter_install_dir }}/postgres_exporter.env
-ExecStart={{ postgres_exporter_binary }} \
-  --web.listen-address=:{{ postgres_exporter_port }} \
-  --collector.stat_statements
-Restart=on-failure
-RestartSec=10s
-
-# Security hardening
-NoNewPrivileges=yes
-PrivateTmp=yes
-
-[Install]
-WantedBy=multi-user.target
@@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key
 ```

 **Manual unseal is required after every pod restart** (node drain, upgrade,
-crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`.
+crash), **per replica** — the server runs three raft peers (see
+`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
+voter nor a standby, so it stays out of the cluster until unsealed:
+`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
+all three. Three peers mean one sealed replica is survivable; two are not.
 Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
 explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
 Secrets keep their last value (they are real k8s Secrets — pods keep running).
@@ -31,8 +31,8 @@ all in state that outlived the VM.
      two hosts every PG is undersized with min_size 2, so one further OSD loss
      blocks IO on every CephFS mount cluster-wide.
 - [ ] Talos metal ISO on a USB stick. Schematic
-      `a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
-      (`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
+      `0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9`
+      (`net.ifnames=1`, intel-ucode, thunderbolt, i915; no qemu-guest-agent).
      Verify by comparing partition offsets and sizes against the ISO with
      `fdisk -l` — blkid labels lie after a rewrite.
 - [ ] Physical access. Secure Boot must be turned off (see below), and that is
@@ -383,6 +383,34 @@ raised to `high_recovery_ops` to speed the evacuation.
 The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
 `ceph config-key rm device/<devid>`; the mgr's devicehealth view may lag.

+### Long-lived TCP sessions to devices outside the cluster die silently
+
+Check this before declaring the conversion done. Anything holding a persistent
+connection to a device on another VLAN — the ESPHome integration is the case
+that bit us — keeps a half-open socket when the node underneath it moves. The
+peer drops its side; the pod's side never learns, because no RST arrives and no
+keepalive fires.
+
+The symptom is that there is no symptom. Home Assistant logged nothing, retried
+nothing, and reported the devices as connected. The devices themselves showed
+"not connected". Hours went into reading logs that were empty by construction.
+
+`/proc/net/tcp` in the pod settles it in one command — `01` is ESTABLISHED,
+`02` is SYN_SENT, and the port is hex:
+
+```sh
+kubectl -n home-assistant exec <ha-pod> -c home-assistant -- \
+  sh -c 'grep ":17A5" /proc/net/tcp'      # 17A5 = 6053, the ESPHome API
+```
+
+Seven `01` entries to devices insisting they were disconnected is the tell. A
+`02` alongside them is a genuinely dead device, not a stale socket — that one
+Home Assistant *was* retrying and *did* log.
+
+Restarting the consuming pod clears it. Reloading just the integration from the
+UI does the same with less disruption. Neither is discoverable from logs, which
+is why it belongs here.
+
 ## Known-stale things that will bite

 - `bootstrap.sh` cannot generate a config at all right now (issue #581), so a
@@ -87,7 +87,7 @@ all three nodes identically from there, since `apply-config --insecure` does
 not care whether the target is a VM or bare metal.

 ```
-https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
+https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.12.11/metal-amd64.iso
 ```

 That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
@@ -149,8 +149,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual**
 (the unseal key lives in `pass`, off-cluster by design).

 ```sh
-# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected).
-kubectl -n openbao get pod openbao-0
+# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
+kubectl -n openbao get pods

 # 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
 #    (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
@@ -158,8 +158,12 @@ kubectl -n openbao get pod openbao-0
 kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
 kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap

-# 3. Unseal with the key from pass (threshold 1).
+# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
+#    A sealed server is neither a raft voter nor a standby, so unsealing only
+#    openbao-0 leaves the three-peer cluster one node short of quorum.
 kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste homelab/openbao/unseal-key
+kubectl -n openbao exec -it openbao-1 -- bao operator unseal   # same key
+kubectl -n openbao exec -it openbao-2 -- bao operator unseal   # same key
 ```

 Verify the store recovers before moving on:
@@ -170,7 +174,9 @@ kubectl -n <ns> get externalsecret            # want: SecretSynced
 ```

 > **Manual unseal is required after every OpenBao pod restart**, not just
-> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`.
+> during rebuilds, and for *each* replica that restarts — a rolling node
+> drain seals all three in turn. Details + rationale:
+> `MIGRATION-sops-openbao.md`.

 ## Step 5 — apps: let the rest reconcile

@@ -37,6 +37,13 @@ resources:
  - esphome-service.yaml
  - esphome-gate-service.yaml
  - esphome-ingress.yaml
+  # mDNS relay — carries IoT-VLAN announcements into HA's zeroconf listener as
+  # unicast, which is the only direction the CNI will move them. Without it HA
+  # never learns that a device's DHCP lease moved. See the header comment in
+  # mdns-relay-deployment.yaml.
+  - mdns-relay-network-attachment.yaml
+  - mdns-relay-service.yaml
+  - mdns-relay-deployment.yaml
  # Zigbee2MQTT — ZHA replacement bridge. Ships scaled to 0; the cutover
  # runbook (zigbee2mqtt-migration-runbook.md) scales it up. The MQTT
  # `zigbee2mqtt` broker user is provisioned by mosquitto-bootstrap.yaml
@@ -0,0 +1,190 @@
+---
+# mDNS relay: gives Home Assistant sight of the IoT VLAN's mDNS announcements
+# without putting a macvlan on the HA pod itself.
+#
+# THE PROBLEM
+#
+# HA's zeroconf listener joins both mDNS groups on eth0 — 224.0.0.251 and
+# ff02::fb — and never hears anything on either.
+# Multicast is link-local, so IoT-VLAN announcements do not reach the pod
+# CIDR; and Cilium runs in tunnel mode (VXLAN) without `enable-multicast`, so
+# it does not replicate multicast between pods either. Both halves of the path
+# are dead, which is why an avahi-style reflector — receive on one interface,
+# re-emit as multicast on the other — cannot work here: the re-emitted packets
+# would be dropped by the CNI.
+#
+# Consequence for HA: the ESPHome integration dials devices at the address
+# cached in its config entry. When a lease rotates, every attempt fails with
+# `[Errno 113] Connect call failed` and the device stays unreachable, because
+# the mDNS announcement that would carry the new address never arrives. New
+# devices are never discovered at all.
+#
+# THE MECHANISM
+#
+# This relay holds the multicast membership on the IoT VLAN through its own
+# macvlan (mdns-relay-network-attachment.yaml) and forwards each datagram it
+# receives to HA as ordinary UNICAST UDP. That is the one direction of travel
+# the CNI will carry.
+#
+# It works because python-zeroconf does not filter on how a packet arrived:
+# its datagram handler validates size, deduplicates, and parses, but performs
+# no check on destination address, source address, or source port. A unicast
+# datagram delivered to :5353 is processed exactly like a multicast one. (mDNS
+# unicast responses are part of RFC 6762 — the QU bit — so this is within the
+# protocol, not a trick.)
+#
+# WHAT THIS DOES AND DOES NOT RESTORE
+#
+# Restored: passive reception. RFC 6762 requires a responder to re-announce
+# when its address changes, so a device that moves to a new lease announces
+# the move, HA sees it, and the ESPHome integration updates the config entry.
+# That is the failure this exists to fix. New devices announcing themselves
+# are likewise seen.
+#
+# NOT restored: HA-initiated browsing. Queries HA sends leave on eth0 as
+# multicast and die there, so HA cannot actively solicit a response. Anything
+# depending on a query rather than an announcement stays broken. Making that
+# work needs a querier on the VLAN side, which this relay deliberately is not.
+#
+# EXPOSURE
+#
+# The receive sockets carry so-bindtodevice=net1, so only datagrams that
+# actually arrived on the IoT VLAN interface are delivered. Binding to net1's
+# address instead is not possible — it is DHCP-leased and unknown at exec time
+# — but binding to the device achieves the same confinement without needing to
+# know the address, and needs no capability: SO_BINDTODEVICE succeeds as an
+# unprivileged uid here.
+#
+# Without it socat would bind the wildcard, and since the namespace CNP grants
+# every pod in home-assistant unrestricted access to every other pod on every
+# port, any sibling pod could reach :5353 over the cluster network and have
+# records forwarded into HA's zeroconf cache.
+#
+# What remains, and cannot be closed here: any host on the IoT VLAN can send
+# this port a datagram. That is the same trust already extended to genuine
+# mDNS from that segment, which any device there can equally forge — the relay
+# widens who can reach HA's listener, not what can be claimed. macvlan traffic
+# bypasses Cilium policy entirely, so the namespace CNP does not constrain
+# this interface either way.
+#
+# Note that the relay is not what makes HA's zeroconf reachable from inside
+# the cluster: HA binds :5353 on the pod address (v4) and the wildcard (v6),
+# and the namespace CNP already permits any sibling pod to reach it directly.
+# Confining the relay removes a second route to that listener; it does not
+# remove the first.
+#
+# Nothing else is exposed: the containers run socat and only socat, with no
+# second listener to reach.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mdns-relay
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: mdns-relay
+spec:
+  # One replica. Extra replicas would deliver a duplicate of every announcement
+  # — harmless, since zeroconf deduplicates, but they buy nothing: this is a
+  # cache-refresh path, and a missed window is corrected by the next
+  # announcement.
+  replicas: 1
+  # Recreate, not RollingUpdate: two relays would briefly hold two DHCP leases
+  # on v30iot for one workload.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: mdns-relay
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mdns-relay
+      annotations:
+        k8s.v1.cni.cncf.io/networks: mdns-relay-iot
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      # One container per address family. mDNS is two protocols — 224.0.0.251
+      # and ff02::fb — and a socket joined to one hears nothing of the other.
+      # HA listens on both (its zeroconf holds both groups on eth0), so a
+      # v4-only relay would silently drop every IPv6 announcement, and devices
+      # that announce only over v6 would stay invisible exactly as before.
+      #
+      # Two processes rather than one: socat carries a single address pair per
+      # invocation, and the families need different receive and send addresses.
+      # They are independent — one crashing does not blind the other — and each
+      # is restarted by the kubelet on its own.
+      containers:
+        # -u is load-bearing on both: unidirectional, left to right. Without it
+        # socat would also carry HA's replies back out, and this must stay a
+        # one-way tap.
+        #
+        # UDP{4,6}-RECV (not -RECVFROM) keeps reading from one socket instead of
+        # forking per datagram — the IoT VLAN's mDNS chatter is continuous, and
+        # a fork per packet would be a process storm.
+        #
+        # The group is joined on net1 specifically, so the relay subscribes on
+        # the IoT VLAN rather than on the pod network.
+        #
+        # Both send targets are the same ClusterIP name, resolved once at start;
+        # UDP4-SENDTO takes its A record and UDP6-SENDTO its AAAA. See
+        # mdns-relay-service.yaml for why that indirection is required and why
+        # the Service is dual-stack.
+        - name: socat-ipv4
+          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
+          imagePullPolicy: IfNotPresent
+          args:
+            - -u
+            - "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr"
+            - "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          # No probes. socat is a single process with no health surface: if it
+          # exits, the container exits and the kubelet restarts it, which is the
+          # entire liveness story. A probe here could only re-assert that PID 1
+          # is running.
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+        # Joining ff02::fb needs no address configuration on net1: the group is
+        # link-local and the kernel's automatic link-local address on the
+        # macvlan is sufficient to receive on it. Whether v30iot hands out
+        # global v6 addresses does not affect reception here.
+        #
+        # ipv6-v6only=1 keeps this socket to v6 only. Both containers share the
+        # pod's network namespace, and a dual-stack v6 socket would overlap the
+        # v4 container's port. Measured behaviour without it was no contention —
+        # the v4 receiver kept taking packets while the v6 one ran — but that
+        # depends on the kernel's bindv6only default rather than on anything
+        # stated here, so the roles are made explicit instead of inherited.
+        - name: socat-ipv6
+          image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
+          imagePullPolicy: IfNotPresent
+          args:
+            - -u
+            - "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr"
+            - "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
@@ -0,0 +1,52 @@
+---
+# mdns-relay's second NIC: macvlan onto v30iot, so the relay has real L2
+# presence on the IoT VLAN and can join both mDNS multicast groups
+# (224.0.0.251 and ff02::fb, port 5353). The v6 group needs no address
+# configuration of its own: it is link-local, and the kernel's automatic
+# link-local address on the macvlan is enough to receive on it, independently
+# of whether v30iot hands out global v6 addresses.
+# Multicast is link-local — it does not cross the L3
+# boundary between the cluster pod CIDR and v30iot — so that membership has to
+# be held by a pod sitting physically on that segment.
+#
+# The address is leased from the UniFi DHCP server on v30iot. Nothing
+# references it: the relay only receives on this interface and sends onward
+# over eth0. See mdns-relay-deployment.yaml for what the relay does with it.
+#
+# Multus's `master` is an interface name. On the k8s-{1,2,3} nodes the v30iot
+# NIC is `ens19` — a stable, busPath-derived predictable name (net.ifnames=1,
+# via the Image Factory schematic), identical across nodes and independent of
+# driver probe order. This is the same master shelly-manager-iot uses.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: mdns-relay-iot
+  namespace: home-assistant
+spec:
+  # Chained CNI, identical in shape to shelly-manager-iot: macvlan leases the
+  # v30iot address by DHCP, then `sbr` confines net1's routes to a
+  # source-based routing table. Without sbr the DHCP-supplied default gateway
+  # (10.2.30.1, option 3) installs a second default route at the same metric
+  # as eth0's, and cluster-internal traffic egresses the IoT VLAN and is
+  # dropped. The relay's forward path to Home Assistant runs over eth0, so
+  # that route has to stay intact.
+  #
+  # Requires the cni-dhcp-daemon DaemonSet in infrastructure/multus.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "mdns-relay-iot",
+      "plugins": [
+        {
+          "type": "macvlan",
+          "master": "ens19",
+          "mode": "bridge",
+          "ipam": {
+            "type": "dhcp"
+          }
+        },
+        {
+          "type": "sbr"
+        }
+      ]
+    }
@@ -0,0 +1,42 @@
+---
+# Stable delivery target for the relay's forwarded mDNS traffic.
+#
+# The relay resolves its send target once, at process start (socat has no
+# re-resolution loop), so it cannot chase the HA pod IP across restarts. A
+# ClusterIP is the fix: the IP is stable for the lifetime of the Service, and
+# Cilium re-DNATs to whatever pod currently backs it.
+#
+# Deliberately a separate Service from `home-assistant`, which targets the
+# nginx proxy sidecar on 8124 and is the XFF trust anchor for the public UI.
+# This one bypasses the sidecar and addresses the home-assistant container's
+# own port directly — containers in a pod share a network namespace, so
+# targetPort 5353 lands on python-zeroconf's listener regardless of which
+# container opened it. Keeping the two apart means the UI path and the
+# discovery path cannot be changed by accident through one another.
+#
+# No CiliumNetworkPolicy change is needed: home-assistant-ns.yaml already
+# permits intra-namespace traffic on all ports, and the relay is in this
+# namespace.
+apiVersion: v1
+kind: Service
+metadata:
+  name: home-assistant-mdns
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: home-assistant
+spec:
+  # Dual-stack, because mDNS is two protocols and HA listens on both: its
+  # zeroconf socket holds 224.0.0.251 on eth0 and ff02::fb alongside it. The
+  # relay forwards each family over its own transport rather than folding v6
+  # announcements into a v4 hop, so the Service has to answer on both.
+  ipFamilyPolicy: PreferDualStack
+  ipFamilies:
+    - IPv4
+    - IPv6
+  selector:
+    app.kubernetes.io/name: home-assistant
+  ports:
+    - name: mdns
+      port: 5353
+      protocol: UDP
+      targetPort: 5353
@@ -12,6 +12,7 @@ resources:
  - prometheusrule-netbird-availability.yaml
  - prometheusrule-netbird-backup.yaml
  - prometheusrule-netbird-ingress.yaml
+  - prometheusrule-openbao.yaml
  - prometheusrule-velero.yaml
  - prometheusrule-woodpecker-rotation.yaml
  # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
@@ -0,0 +1,91 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: openbao
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: openbao.rules
+      rules:
+        # WHY THIS WATCHES POD READINESS RATHER THAN A SEAL METRIC
+        #
+        # OpenBao is not scraped: there is no ServiceMonitor for it, and its
+        # metrics endpoint needs either a token or an explicit
+        # `unauthenticated_metrics_access` telemetry setting, neither of which
+        # is configured. Confirmed against Prometheus — no bao_* or vault_*
+        # series exist. External Secrets exports nothing usable either, so the
+        # ClusterSecretStore going NotReady is invisible as a metric.
+        #
+        # kube-state-metrics readiness carries the signal anyway, and does so
+        # for a structural reason rather than by luck: a sealed OpenBao fails
+        # its readiness probe, because a sealed server can neither serve
+        # requests nor act as a raft standby. Sealed is therefore always
+        # not-Ready. The converse does not hold — a crash or a stuck
+        # scheduling would also fire this — but both want a human too, so the
+        # imprecision costs nothing.
+        - alert: OpenBaoReplicaNotReady
+          # A replica that is sealed never recovers on its own, so the
+          # duration is only there to ignore ordinary rescheduling: a healthy
+          # pod moving between nodes is Ready again in well under a minute.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "OpenBao has {{ $value }} of 3 replicas ready"
+            description: >-
+              An OpenBao replica has been unavailable for 10 minutes. The usual
+              cause is a sealed replica after a node drain — every restart
+              seals it, and it stays out of the raft cluster until unsealed by
+              hand. The service is still up on the remaining replicas, so
+              nothing is broken yet, but redundancy is reduced and a second
+              drain would cost quorum. Check `kubectl -n openbao get pods` and
+              `bao status` in the not-ready pod; unseal with the key from
+              `pass` (homelab/openbao/unseal-key).
+
+        - alert: OpenBaoQuorumLost
+          # Raft quorum of a three-node cluster is two. At one ready replica
+          # the cluster cannot elect a leader and the secret store is down:
+          # the ClusterSecretStore goes InvalidProviderConfig and no
+          # ExternalSecret can refresh.
+          #
+          # Shorter window than the warning above because this is an outage,
+          # not a degradation — but not zero, so a rolling restart that
+          # briefly dips does not page.
+          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenBao below raft quorum — {{ $value }} of 3 replicas ready"
+            description: >-
+              Fewer than two OpenBao replicas are ready, so raft cannot hold
+              quorum and the secret store is unavailable. ExternalSecrets stop
+              refreshing; existing Secrets keep their last values, so failures
+              surface only when something rotates or a new secret is needed.
+              Unseal the sealed replicas (key in `pass`,
+              homelab/openbao/unseal-key) — each one individually, since a
+              sealed server cannot be a standby.
+
+        - alert: OpenBaoMetricsMissing
+          # Guards the two alerts above. Both are threshold comparisons, so if
+          # kube-state-metrics stops exporting this series — the StatefulSet
+          # deleted, the namespace gone, kube-state-metrics itself broken —
+          # they silently stop evaluating and the silence looks identical to
+          # health. This makes that state say something.
+          expr: absent(kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"})
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No readiness metric for the OpenBao StatefulSet"
+            description: >-
+              kube-state-metrics has not reported
+              kube_statefulset_status_replicas_ready for the openbao
+              StatefulSet in 15 minutes. Either the StatefulSet is gone or
+              kube-state-metrics is not scraping — and while this is absent,
+              OpenBaoReplicaNotReady and OpenBaoQuorumLost cannot fire at all.
@@ -275,8 +275,11 @@ spec:
          - alertmanager-smtp
      # SMTP via Proton submission (smtp.protonmail.ch:587 STARTTLS).
      # The Proton SMTP token is a per-account credential generated in
-      # the Proton webapp (Settings -> IMAP/SMTP) and lives in the
-      # alertmanager-smtp SOPS-encrypted Secret. Outbound 587 from the
+      # the Proton webapp (Settings -> IMAP/SMTP). It is stored in
+      # OpenBao at apps/monitoring/alertmanager-smtp and projected into
+      # the alertmanager-smtp Secret by
+      # ./alertmanager-smtp-externalsecret.yaml — rotating the token
+      # means updating OpenBao, not this repo. Outbound 587 from the
      # monitoring ns is not blocked by any CNP today.
      config:
        global:
@@ -55,7 +55,7 @@ spec:
        # stdlib only; Flux keeps the tag in lockstep via the shared image
        # policy) — no extra dependency.
        - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
          command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
          volumeMounts:
            - name: data
@@ -65,7 +65,7 @@ spec:
              readOnly: true
      containers:
        - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.9.13 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.0 # {"$imagepolicy": "flux-system:music-assistant"}
          env:
            - name: LOG_LEVEL
              value: info
@@ -4,3 +4,12 @@ resources:
  - cni-plugins-installer.yaml
  - cni-dhcp-daemon.yaml
  - multus-daemonset.yaml
+  # Whereabouts — cluster-coordinated IPAM, for attachments on segments with
+  # no DHCP server. CRDs first: the DaemonSet's control loop reads them at
+  # startup. See the header in whereabouts-daemonset.yaml for why neither
+  # dhcp nor host-local covers the Ceph SAN.
+  - whereabouts-crd-ippools.yaml
+  - whereabouts-crd-overlappingrangeipreservations.yaml
+  - whereabouts-crd-nodeslicepools.yaml
+  - whereabouts-daemonset.yaml
+  - whereabouts-reconciler.yaml
@@ -0,0 +1,71 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: ippools.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: IPPool
+    listKind: IPPoolList
+    plural: ippools
+    singular: ippool
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: IPPool is the Schema for the ippools API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: IPPoolSpec defines the desired state of IPPool
+            properties:
+              allocations:
+                additionalProperties:
+                  description: IPAllocation represents metadata about the pod/container
+                    owner of a specific IP
+                  properties:
+                    id:
+                      type: string
+                    ifname:
+                      type: string
+                    podref:
+                      type: string
+                  required:
+                  - id
+                  - podref
+                  type: object
+                description: |-
+                  Allocations is the set of allocated IPs for the given range. Its` indices are a direct mapping to the
+                  IP with the same index/offset for the pool's range.
+                type: object
+              range:
+                description: Range is a RFC 4632/4291-style string that represents
+                  an IP address and prefix length in CIDR notation
+                type: string
+            required:
+            - allocations
+            - range
+            type: object
+        type: object
+    served: true
+    storage: true
@@ -0,0 +1,79 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: nodeslicepools.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: NodeSlicePool
+    listKind: NodeSlicePoolList
+    plural: nodeslicepools
+    singular: nodeslicepool
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: NodeSlicePool is the Schema for the nodesliceippools API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: NodeSlicePoolSpec defines the desired state of NodeSlicePool
+            properties:
+              range:
+                description: |-
+                  Range is a RFC 4632/4291-style string that represents an IP address and prefix length in CIDR notation
+                  this refers to the entire range where the node is allocated a subset
+                type: string
+              sliceSize:
+                description: SliceSize is the size of subnets or slices of the range
+                  that each node will be assigned
+                type: string
+            required:
+            - range
+            - sliceSize
+            type: object
+          status:
+            description: NodeSlicePoolStatus defines the desired state of NodeSlicePool
+            properties:
+              allocations:
+                description: Allocations holds the allocations of nodes to slices
+                items:
+                  properties:
+                    nodeName:
+                      description: NodeName is the name of the node assigned to this
+                        slice, empty node name is an available slice for assignment
+                      type: string
+                    sliceRange:
+                      description: SliceRange is the subnet of this slice
+                      type: string
+                  required:
+                  - nodeName
+                  - sliceRange
+                  type: object
+                type: array
+            required:
+            - allocations
+            type: object
+        type: object
+    served: true
+    storage: true
@@ -0,0 +1,57 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.14.0
+  name: overlappingrangeipreservations.whereabouts.cni.cncf.io
+spec:
+  group: whereabouts.cni.cncf.io
+  names:
+    kind: OverlappingRangeIPReservation
+    listKind: OverlappingRangeIPReservationList
+    plural: overlappingrangeipreservations
+    singular: overlappingrangeipreservation
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: OverlappingRangeIPReservation is the Schema for the OverlappingRangeIPReservations
+          API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: OverlappingRangeIPReservationSpec defines the desired state
+              of OverlappingRangeIPReservation
+            properties:
+              containerid:
+                type: string
+              ifname:
+                type: string
+              podref:
+                type: string
+            required:
+            - podref
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
@@ -0,0 +1,196 @@
+---
+# Whereabouts: cluster-wide IPAM for Multus attachments.
+#
+# WHY THIS EXISTS
+#
+# The attachments in this cluster lease their addresses by DHCP
+# (cni-dhcp-daemon.yaml). That works where a DHCP server is on the wire —
+# v30iot has one. It does not work on the Ceph SAN (VLAN 49), which carries
+# Ceph-internal traffic and nothing else, by design: there is no DHCP server
+# there and there must not be one.
+#
+# `host-local`, already installed by cni-plugins-installer.yaml, cannot stand
+# in. It keeps its allocation state per node with no cluster-wide
+# coordination, so on a shared L2 segment two nodes hand out the same address.
+# Whereabouts coordinates through the Kubernetes API instead — allocations
+# live in the ippools/overlappingrangeipreservations CRDs, so nothing about
+# the address assignment touches the segment being addressed. That property is
+# what makes it usable on an isolated SAN.
+#
+# HOST ACCESS
+#
+# The DaemonSet runs with hostNetwork and privileged. That is how upstream
+# ships it, and it is what the three CNI components already here do —
+# multus-daemonset.yaml, cni-dhcp-daemon.yaml and cni-plugins-installer.yaml
+# all carry both. The reason is the same in each case: the pod writes a CNI
+# binary and its config into /opt/cni/bin and /etc/cni/net.d on the node,
+# which is node plumbing rather than a workload, and it must work before the
+# attachment it provides addresses for exists.
+#
+# The reconciler is deliberately NOT in here. It needs neither, and runs as an
+# ordinary pod-network Deployment — see whereabouts-reconciler.yaml.
+#
+# Vendored from upstream v0.9.4 (doc/crds/daemonset-install.yaml), with the
+# image tag pinned; upstream ships `:latest`. The CRDs it needs are the three
+# whereabouts-crd-*.yaml files alongside this one, vendored unmodified so an
+# upstream diff stays readable.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: whereabouts
+  namespace: kube-system
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: whereabouts
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: whereabouts-cni
+subjects:
+  - kind: ServiceAccount
+    name: whereabouts
+    namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: whereabouts-cni
+rules:
+  - apiGroups:
+      - whereabouts.cni.cncf.io
+    resources:
+      - ippools
+      - overlappingrangeipreservations
+      - nodeslicepools
+    verbs:
+      - get
+      - list
+      - watch
+      - create
+      - update
+      - patch
+      - delete
+  - apiGroups:
+      - coordination.k8s.io
+    resources:
+      - leases
+    verbs:
+      - '*'
+  - apiGroups: [""]
+    resources:
+      - pods
+    verbs:
+      - list
+      - watch
+      - get
+  - apiGroups: [""]
+    resources:
+      - nodes
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - network-attachment-definitions
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ""
+      - events.k8s.io
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+      - update
+      - get
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: whereabouts-config
+  namespace: kube-system
+  annotations:
+    kubernetes.io/description: |
+      Configmap containing user customizable cronjob schedule
+data:
+  # Read by the reconciler (whereabouts-reconciler.yaml), not by this
+  # DaemonSet. Governs how often stale allocations are swept — the case that
+  # matters here is a node lost hard enough that the CNI DEL never ran, which
+  # otherwise leaves an address held by a pod that no longer exists.
+  cron-expression: "30 4 * * *"
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: whereabouts
+  namespace: kube-system
+  labels:
+    tier: node
+    app: whereabouts
+spec:
+  selector:
+    matchLabels:
+      name: whereabouts
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: whereabouts
+        name: whereabouts
+    spec:
+      hostNetwork: true
+      serviceAccountName: whereabouts
+      # Every node here is a control plane node, so the NoSchedule toleration
+      # is not optional — without it this schedules nowhere.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: whereabouts
+          command: ["/bin/sh"]
+          args:
+            - -c
+            - |
+              SLEEP=false source /install-cni.sh
+              /token-watcher.sh &
+              /ip-control-loop -log-level debug
+          image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
+          env:
+            - name: NODENAME
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: spec.nodeName
+            - name: WHEREABOUTS_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "100Mi"
+            limits:
+              cpu: "100m"
+              memory: "200Mi"
+          securityContext:
+            privileged: true
+          volumeMounts:
+            - name: cnibin
+              mountPath: /host/opt/cni/bin
+            - name: cni-net-dir
+              mountPath: /host/etc/cni/net.d
+      volumes:
+        - name: cnibin
+          hostPath:
+            path: /opt/cni/bin
+        - name: cni-net-dir
+          hostPath:
+            path: /etc/cni/net.d
@@ -0,0 +1,75 @@
+---
+# Sweeps whereabouts allocations that no live pod holds any more.
+#
+# The normal release path is the CNI DEL when a pod goes away. It does not run
+# when a node is lost hard — power cut, kernel panic, a reboot that never
+# comes back — and the allocation then stays claimed by a pod that no longer
+# exists. On a small range that matters: the Ceph SAN is a /29, so a handful
+# of leaked addresses is the difference between an OSD getting an address and
+# not.
+#
+# Runs as an ordinary pod-network Deployment. Unlike the DaemonSet next to it,
+# this needs no host access at all — it only talks to the Kubernetes API,
+# where the allocations live.
+#
+# replicas stays at 1: there is no leader election, so a second replica would
+# reconcile the same records concurrently.
+#
+# Vendored from upstream v0.9.4 (doc/crds/reconciler-deployment.yaml) with the
+# image tag pinned; upstream ships `:latest`. The schedule comes from the
+# whereabouts-config ConfigMap in whereabouts-daemonset.yaml.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: whereabouts-reconciler
+  namespace: kube-system
+  labels:
+    tier: node
+    app: whereabouts-reconciler
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      name: whereabouts-reconciler
+  template:
+    metadata:
+      labels:
+        tier: node
+        app: whereabouts-reconciler
+        name: whereabouts-reconciler
+    spec:
+      serviceAccountName: whereabouts
+      # Every node here is a control plane node.
+      tolerations:
+        - operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: whereabouts-reconciler
+          command: ["/ip-reconciler"]
+          args:
+            - -log-level
+            - debug
+          image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
+          env:
+            - name: WHEREABOUTS_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "50Mi"
+            limits:
+              cpu: "100m"
+              memory: "50Mi"
+          volumeMounts:
+            - name: cron-scheduler-configmap
+              mountPath: /cron-schedule
+      volumes:
+        - name: cron-scheduler-configmap
+          configMap:
+            name: whereabouts-config
+            defaultMode: 0744
+            items:
+              - key: "cron-expression"
+                path: "config"
@@ -42,12 +42,28 @@ spec:
      # the agent-injector mutating webhook is unused surface.
      enabled: false
    server:
-      # Single-node raft (not standalone/file) so we get `bao operator raft
-      # snapshot` as a consistent backup primitive. TLS terminates at the
-      # Traefik ingress; in-cluster traffic is plain HTTP.
+      # Raft (not standalone/file) for two reasons: `bao operator raft
+      # snapshot` as a consistent backup primitive, and real HA across three
+      # nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
+      # plain HTTP.
      ha:
        enabled: true
-        replicas: 1
+        # Three, not two. Raft needs a quorum, and the quorum of two is two —
+        # a two-replica cluster tolerates no failures at all and would stop
+        # the moment one pod is sealed or rescheduled, which is worse than
+        # the single replica it replaced. Three tolerates one.
+        #
+        # What this buys, and what it does not: a node drain seals the replica
+        # on it, and raft elects a new leader from the remaining standbys
+        # within seconds, so the service does not go down. It does NOT remove
+        # the manual unseal — a sealed server can neither serve requests nor
+        # act as standby, so that replica stays out until someone unseals it.
+        #
+        # The consequence for rolling node maintenance is worth stating: a
+        # Talos upgrade drains all three nodes in turn, sealing each replica
+        # as it goes. Unseal each one as it comes back, or the third drain
+        # takes the last unsealed replica with it and quorum is lost anyway.
+        replicas: 3
        raft:
          enabled: true
          setNodeId: true
@@ -65,8 +81,32 @@ spec:
              cluster_address = "[::]:8201"
            }

+            # retry_join is what makes a replica join the existing raft
+            # cluster instead of coming up as an isolated, uninitialized
+            # node. Without it, scaling the StatefulSet produces pods that
+            # never become peers and would each need a manual
+            # `bao operator raft join`.
+            #
+            # All three peers are listed on every node, including the node
+            # itself. That is deliberate: the stanza is a list of possible
+            # leaders, a node ignores its own entry, and listing all three
+            # means the set is identical on every replica and stays correct
+            # regardless of which pod currently leads.
+            #
+            # Addressed through the headless service, so each entry resolves
+            # to one specific pod rather than load-balancing across them.
            storage "raft" {
              path = "/openbao/data"
+
+              retry_join {
+                leader_api_addr = "http://openbao-0.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-1.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-2.openbao-internal:8200"
+              }
            }

            service_registration "kubernetes" {}
@@ -53,7 +53,13 @@ spec:
              command: ["/bin/sh", "-ec"]
              args:
                - |
-                  export BAO_ADDR=http://openbao.openbao.svc.cluster.local:8200
+                  # openbao-active, not openbao: with more than one replica
+                  # the plain service selects any pod, and a raft snapshot has
+                  # to be taken against the leader. Standbys forward ordinary
+                  # requests, but this is not one to leave to chance — the
+                  # active service selects the leader by label, so the
+                  # snapshot is always taken where it is consistent.
+                  export BAO_ADDR=http://openbao-active.openbao.svc.cluster.local:8200
                  BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
                    role=openbao-snapshot \
                    jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")
@@ -8,6 +8,17 @@ spec:
    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
+  # Do not start restarting OSDs while PGs are still recovering. The default
+  # is false, so Rook would proceed regardless — and paired with the old
+  # osdMaxUpdatesInParallel default that means it can take the next OSD down
+  # while the previous one's data is still being caught up.
+  #
+  # With three OSDs and size 3 there is no spare copy to fall back on: the
+  # gate between one OSD returning and the next going down is the only thing
+  # that keeps a rolling change from becoming an outage. See
+  # storage.osdMaxUpdatesInParallel below, which bounds the same operation
+  # from the other side.
+  upgradeOSDRequiresHealthyPGs: true
  mon:
    count: 3
    allowMultiplePerNode: false
@@ -36,26 +47,27 @@ spec:
  # (paperless + audiobookshelf + immich + ... all clone at once).
  cephConfig:
    mon:
-      # Ceph's 30% default assumes the mon has a filesystem to itself, where
-      # 30% free is a meaningful reserve. Here the mons live on
-      # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB
-      # of containerd image and snapshot storage — and the mon DB itself is
-      # ~106 MB. So the default measures container churn, not Ceph: k8s-2
-      # trips MON_DISK_LOW whenever the image cache fills, and image GC does
-      # not clear it (most of /var/lib/containerd is overlayfs snapshots held
-      # by running containers, which GC cannot reclaim — see #434).
-      # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database.
-      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
-      # monitoring-rules watches /var directly for the real risk, which is the
-      # node filling rather than the mon growing.
+      # The Ceph default, stated rather than left out. Removing the line would
+      # not restore it: `ceph config` keeps a value that only disappears from
+      # this file, so the old 20 would sit in the mon config database
+      # indefinitely — the same trap the bluestore settings below describe,
+      # and the one that crash-looped an OSD during the network provider
+      # change when public_network survived being deleted here.
      #
-      # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and
-      # the default 30% becomes meaningful again, at which point this override
-      # should be dropped rather than carried over. It cannot be done on the
-      # current cluster — every StorageClass is Ceph-backed, so a mon PVC
-      # would be circular, and Rook only applies mon.volumeClaimTemplate to
-      # newly created mons, never migrating existing ones.
-      mon_data_avail_warn: "20"
+      # It was 20 because the mons share the node's /var with containerd, and
+      # on the VMs that was ~60 GB: 30% free measured image and snapshot churn
+      # rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever
+      # the image cache filled and GC could not reclaim it (#434).
+      #
+      # Bare metal removed the premise. /var is now ~931 GB per node at 7-8%
+      # used, so 30% free is a real reserve again and the ~106 MB mon database
+      # is irrelevant against it. Reaching 70% would be a genuine signal about
+      # the node rather than noise about the image cache.
+      #
+      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
+      # monitoring-rules still watches /var directly for the real risk, which
+      # is the node filling rather than the mon growing.
+      mon_data_avail_warn: "30"
      # Stop compacting the whole RocksDB prefix on every paxos trim.
      #
      # Upstream default is true — "Compact a certain prefix (including paxos)
@@ -147,19 +159,65 @@ spec:
    ssl: false
  monitoring:
    enabled: true
-  # Dedicated Ceph cluster network on VLAN 49 / 10.99.0.0/29, carried by an
-  # X710 SFP+ port PCI-passed-through to each Talos K8s VM (see
-  # talos/patches/k8s-{1,2,3}.yaml and opentofu/vm_15{0,1,2}_k8s_*.tf).
-  # `addressRanges` requires `provider: host` — Rook v1.18 explicitly notes
-  # the default (CNI) provider ignores addressRanges. Public range matches
-  # the existing v40srv subnet that the controlplane VMs already live on.
+  # Dedicated Ceph cluster network on VLAN 49, carried by the second X710
+  # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus.
+  #
+  # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK
+  #
+  # Because it reaches it for every daemon, not just the ones that need it.
+  # The cluster network is an OSD concern: per the Ceph documentation it
+  # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always
+  # operate on the public network". `addressRanges` is only honoured under
+  # provider: host, so using it put the mons on host networking as a side
+  # effect — making a mon's address the node's DHCP lease.
+  #
+  # That is not theoretical. When a lease moved, the mon could no longer bind
+  # ("Cannot assign requested address") and crash-looped. Rook detected it and
+  # started a failover, but with three mons on three nodes each holding the
+  # mon host ports, the replacement could not be scheduled anywhere — the
+  # canary sat Pending on "didn't have free ports". Two of three nodes hit
+  # this during the Talos v1.12.11 upgrade and both needed hand intervention.
+  #
+  # Only `cluster` is selected here. With `public` left unset Rook puts client
+  # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep
+  # Rook-managed identities and hold no host ports — which is what removes
+  # that failure mode rather than working around it.
+  #
+  # ISOLATION IS PRESERVED
+  #
+  # Rook's Multus prerequisites about hosts routing to the Multus network
+  # apply to a Multus *public* network. Selecting only `cluster` means they do
+  # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing
+  # else, no router, no other participants. After this settles, the node
+  # addresses on that segment (10.99.0.1-.3) have no remaining purpose and
+  # should come out of the Talos configs — at which point the attachment's
+  # pool can take the whole /29.
+  #
+  # Reached from provider: "" rather than directly from host, which Rook's CRD
+  # rejects. See the phase A commit.
+  #
+  # ONE-TIME STEP THAT THIS FILE CANNOT EXPRESS, recorded because it is not
+  # reproducible from the manifests alone. Removing addressRanges here does
+  # not clear public_network and cluster_network from Ceph's own config
+  # database — the same trap the bluestore comment above describes: `ceph
+  # config` retains a value that is only deleted from this file. During the
+  # phase A rollout the first OSD came up on the pod network, still had
+  # public_network 10.2.40.0/24 in the config database, found no interface in
+  # it and crash-looped on "Failed to pick public address". Both keys were
+  # removed with `ceph config rm global public_network` and the same for
+  # cluster_network; Rook then wrote cluster_network afresh from the
+  # attachment below. Anyone rebuilding this cluster from scratch will not hit
+  # it — the config database starts empty — but anyone repeating a provider
+  # change on a live cluster will.
+  #
+  # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel
+  # above: one OSD restarts at a time and only once PGs are clean again.
+  #
+  # The attachment itself is san-network-attachment.yaml.
  network:
-    provider: host
-    addressRanges:
-      public:
-        - "10.2.40.0/24"
-      cluster:
-        - "10.99.0.0/29"
+    provider: multus
+    selectors:
+      cluster: rook-ceph/ceph-san
  # Nodes are listed explicitly rather than useAllNodes, so that a node can be
  # taken out of the storage fleet by removing it here. Under useAllNodes the
  # operator re-provisions an OSD on any node whose device still carries a
@@ -193,6 +251,15 @@ spec:
  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
  #   S437NF0M501176    PM983     — this OSD
  storage:
+    # One OSD restarted at a time. The default is 20, which on a three-OSD
+    # cluster means all of them at once — every PG inactive, every pool
+    # unreachable, for as long as the restart takes. The default suits a
+    # cluster large enough that 20 OSDs is a small fraction of it; here it is
+    # the whole thing.
+    #
+    # This binds any change that rewrites the OSD deployment spec, not just
+    # Ceph version upgrades — a network provider change is one such.
+    osdMaxUpdatesInParallel: 1
    useAllNodes: false
    useAllDevices: false
    nodes:
@@ -1,6 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  # Attachment for the VLAN 49 cluster network. Inert until cephcluster.yaml
+  # names it under network.selectors.cluster — creating it changes nothing on
+  # its own.
+  - san-network-attachment.yaml
  - cephcluster.yaml
  - toolbox.yaml
  - storageclass.yaml
@@ -0,0 +1,84 @@
+---
+# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
+# without every Ceph daemon being put on host networking to get there.
+#
+# WHAT THIS REPLACES, AND WHY
+#
+# `addressRanges` only takes effect under `provider: host` — Rook ignores it
+# for the default CNI provider. So carrying replication over the dedicated
+# 10G link meant setting provider: host, which is not per-daemon: it puts
+# mons on host networking too. A mon's address is then the node's DHCP lease,
+# and when that lease moves the mon cannot bind and fails. Worse, three mons
+# on three nodes each hold the mon host ports, so Rook cannot schedule a
+# replacement anywhere and the failover it starts cannot finish.
+#
+# None of that has anything to do with the goal. Per the Ceph documentation
+# the cluster network carries OSD heartbeat, replication and recovery only —
+# "Ceph Monitors always operate on the public network". Attaching it here
+# gives the OSDs the link and leaves everything else on the pod network,
+# where Rook manages identity itself.
+#
+# ISOLATION
+#
+# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
+# are deliberate:
+#
+#   No gateway and no routes. There is no router on this segment and there
+#   must not be one. whereabouts adds neither unless asked, so nothing here
+#   can leak onto or off the SAN. This also means no `sbr` chaining is
+#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
+#   route to confine, because there is no DHCP and no default route.
+#
+#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
+#   migration ran, because the nodes still held .1-.3 for as long as Ceph
+#   needed host networking to reach this link — three addresses for three
+#   OSDs, no spare. The nodes have since given those up (the link is only a
+#   macvlan parent now and needs no address of its own), so the reservation
+#   has nothing left to avoid.
+#
+#   Keeping the pool inside the /29 rather than moving it elsewhere on the
+#   segment is not cosmetic. There is no router here, so an address outside
+#   the /29 would be unreachable from anything still using one inside it —
+#   which is exactly what would break a future rolling change, where migrated
+#   and un-migrated OSDs have to keep talking to each other.
+#
+#   Six addresses for three OSDs is the headroom the migration could not
+#   afford: room for a leaked allocation to sit until the reconciler collects
+#   it, and room to add a second OSD per node without touching this file.
+#
+# IPAM
+#
+# whereabouts, because neither alternative works on this segment: dhcp needs
+# a server that must not exist here, and host-local keeps its state per node
+# with no cluster-wide coordination, so two nodes would hand out the same
+# address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
+#
+# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
+# it is stated so that a change to the parent does not silently halve
+# replication throughput without anything in the tree recording the
+# expectation.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: ceph-san
+  namespace: rook-ceph
+spec:
+  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
+  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
+  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
+  # reboots.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "ceph-san",
+      "type": "macvlan",
+      "master": "enp2s0f1np1",
+      "mode": "bridge",
+      "mtu": 9000,
+      "ipam": {
+        "type": "whereabouts",
+        "range": "10.99.0.0/29",
+        "range_start": "10.99.0.1",
+        "range_end": "10.99.0.6"
+      }
+    }
@@ -20,16 +20,30 @@ machine:
  #                            the MS-01 mesh later needs neither a new image
  #                            nor a reinstall — which is exactly the property
  #                            lost by installing without it.
+  #   siderolabs/i915          Intel GPU microcode and kernel modules for the
+  #                            Alder Lake-P Iris Xe in the MS-01. No workload
+  #                            requests /dev/dri today; it is here for the same
+  #                            reason as thunderbolt, so that wanting hardware
+  #                            transcoding later is a config change rather than
+  #                            another rolling upgrade of every node.
  #
-  # a21cec8f… is the bare-metal schematic described in RUNBOOK-cold-rebuild.md:
-  # same net.ifnames=1 as the Proxmox one (the predictable interface names the
-  # per-node VLAN parents match against), minus qemu-guest-agent.
+  # Not included, and why, so the absences are not mistaken for oversights:
+  # the X710 (i40e) and I226 (igc) NICs need no firmware extension —
+  # intel-ice-firmware covers E810/ice only. mei is a prerequisite for Intel
+  # Arc discrete GPUs, of which there are none. iscsi-tools, nvme-cli and
+  # util-linux-tools have no consumer: storage is Ceph RBD, and device health
+  # is reported by Ceph itself rather than a node-level exporter.
+  #
+  # 0f6fb8bd… is the bare-metal schematic: same net.ifnames=1 as the Proxmox
+  # one (the predictable interface names the per-node VLAN parents match
+  # against), minus qemu-guest-agent. It supersedes a21cec8f…, which was
+  # identical but without i915 — see RUNBOOK-cold-rebuild.md.
  #
  # Only read at install and upgrade time. Changing it here does nothing to a
  # running node until that node is upgraded. Bump the tag with the Talos
  # version — it must match the running version, or the upgrade is a downgrade.
  install:
-    image: factory.talos.dev/metal-installer/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64:v1.12.6
+    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.12.11
  features:
    rbac: true
    stableHostname: true
@@ -20,6 +20,29 @@ machine:
    # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
    # This disables v6 entirely on that link, including link-local.
    net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument, applied to the physical ports rather than a VLAN child.
+    #
+    # enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and
+    # enp87s0 is unused. Both sit on the untagged fallback network, where by
+    # design nothing is supposed to run — and both were autoconfiguring
+    # themselves from its router advertisements: an fd00::/64 ULA each, plus
+    # an IPv6 default route each.
+    #
+    # The result was four IPv6 default routes per node at identical priority
+    # 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a
+    # globally routable source address. With nothing to break the tie,
+    # outbound IPv6 mostly left through an interface whose only usable source
+    # was a ULA, so the cluster had no IPv6 egress at all while IPv4 worked
+    # normally. It surfaced as two NetBird probes reporting the proxies down
+    # for 30 hours; the proxies were fine and answered IPv6 in 13 ms from
+    # outside.
+    #
+    # Verified on k8s-1 before rolling to the others: disabling v6 on a VLAN
+    # parent leaves the VLAN child alone. mgmt0 kept its global address and
+    # its fd40:: address, Ceph did not notice, and every failing probe
+    # recovered on the next scrape.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
    # Three NVMe drives are ordinary block devices to Talos here: the Kingston
    # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
@@ -94,6 +117,30 @@ vlanID: 40
 parent: enp2s0f0np0
 up: true
 ---
+# Management addressing is DHCP and the lease is NOT reserved anywhere in this
+# repo — the k8s-{1,2,3}.srv.renner.uno reservations live on the UDM out of band
+# and opentofu/unifi.tf declares no live resources. The addresses do move under
+# a running node: k8s-3 went 10.2.40.168 -> .138 mid-flight on 2026-08-26
+# without its Node object ever re-registering (issue #635).
+#
+# kube-apiserver does not follow when that happens. Talos starts it with
+# `--advertise-address=$(POD_IP)` and no `--endpoint-reconciler-type` override,
+# so the advertised address is fixed at pod start and the default lease
+# reconciler goes on renewing the masterlease under the OLD address. Nothing
+# expires it while the process lives, so `default/kubernetes` keeps handing out
+# a dead endpoint and the node's live address is never registered at all — 19h
+# of TargetDown on job="apiserver", a monitoring blind spot on one of three
+# apiservers, and it cleared only when the static pod restarted.
+#
+# So after any node re-IPs, check that
+# `kubectl get endpoints kubernetes -n default` agrees with
+# `kubectl get nodes -o wide`, and restart the offending node
+# (`talosctl -n <node> reboot`) if it does not.
+#
+# Pinning mgmt0 with a static AddressConfig alongside this document would remove
+# the trigger, but it needs an address outside the UDM's DHCP pool — which is
+# not declared in this repo — and §6 of RUNBOOK-baremetal-conversion.md applies:
+# prove it with `talosctl apply-config --mode=try` before making it permanent.
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
@@ -117,13 +164,20 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
-# segment, and deliberately has no default route.
+# untagged on the wire. Carries OSD<->OSD replication and nothing else.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.1/29, because reaching this link required
+# CephCluster.spec.network.provider: host, and that put every Ceph daemon on
+# host networking, mons included. Nothing needs it now, and by the rule this
+# segment is built on — Ceph-internal traffic and nothing else — the node has
+# no business on it either.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.1/29
@@ -9,6 +9,15 @@ machine:
    # macvlan parent for the v30iot pod attachments; host addresses on that L2
    # make node replies bypass the UDM and break every UDM-DNATed flow.
    net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
    # Three NVMe drives are visible on bare metal. Select the installer target
    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -89,10 +98,17 @@ up: true
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
 # untagged on the wire. OSD<->OSD replication only. Not routed off its own
 # segment and deliberately has no default route.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.2/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.2/29
@@ -8,6 +8,15 @@ machine:
  sysctls:
    # v30iot NIC carries no host addressing — see k8s-1.yaml.
    net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
    # Three NVMe drives are visible on bare metal. Select the installer target
    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -73,10 +82,16 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml.
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.3/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.3/29
@@ -1,27 +0,0 @@
-# k8s-3 was the last Proxmox VM. neptune now runs Talos on bare metal, and its
-# disks were wiped during the conversion, so the VM this module described does
-# not exist in any form.
-#
-# `removed` rather than simply deleting vm_152_k8s_3.tf: dropping the resource
-# from configuration alone makes OpenTofu plan a DESTROY, and planning a destroy
-# refreshes the resource through the bpg/proxmox provider — which now has no
-# endpoint to reach. That is exactly what has been failing every infrastructure
-# plan since the conversion:
-#
-#   Error: failed to get resources list of type ("vm") for cluster:
-#   dial tcp 10.2.40.201:8006: connect: no route to host
-#     with module.k8s_3.proxmox_virtual_environment_vm.this
-#
-# `lifecycle { destroy = false }` forgets the state entry without attempting to
-# contact Proxmox. Nothing is orphaned by doing so: the guest, its disks and the
-# hypervisor underneath it are all already gone.
-#
-# This block can be deleted once one plan has run cleanly with it — it is a
-# one-shot instruction, not standing configuration.
-removed {
-  from = module.k8s_3
-
-  lifecycle {
-    destroy = false
-  }
-}