The stack's first PR turned pruning off in infrastructure,
infrastructure-crds, home-assistant, authentik, immich, matrix, paperless,
trmnl and woodpecker, so that the moved objects were dropped from their
inventories instead of deleted. Merged after every move has reconciled,
this turns it back on: the moved objects are no longer in the old
Kustomizations' inventories by then, so garbage collection has nothing of
theirs to delete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restore rehearsal: a throwaway three-node cluster on Hetzner Cloud that
runs the production tree and restores production's backups into itself.
hcloudtest (bootstrap.sh) runs SITE=hcloud on cpx52, and nothing stands in
for the home network:
- site-hcloud.yaml states the Hetzner site's facts, as site-homelab.yaml
does at home: etcd advertised on the node segment 10.90.0.0/16; a
cilium-site ConfigMap with tunnel (VXLAN) routing over the node segment
and only the pod CIDRs exempt from masquerading (Hetzner networks are
layer 3, IPv4-only, and drop packets with a source not assigned to the
sending server); a ResolverConfig with k8s-gateway first and a
k8s-gateway-site ConfigMap pointing at Hetzner's resolvers. The node
range is read from the machine document only, now that the patch holds
more than one (run 28).
- hcloud-1..3.yaml put a 345GiB OSD partition beside a 60GiB EPHEMERAL on
the system disk: the project's volume quota cannot hold three OSD
volumes. No Hetzner volume is created for a raw volume on the system disk.
- The Talos snapshot is built when missing (a rescue-mode server writes the
Image Factory image; removed on failure), matched by version and
schematic. This host's inbound firewall replaces earlier runs' without a
server ever being left without one. The Hetzner token comes from OpenBao
(homelab-backup-verify project). Access files land in
~/.local/state/homelab-hcloudtest instead of access-staging.
- Containment: an outbound firewall on the nodes, applied before any
config, admits udp and icmp anywhere and tcp to the complement of
proxy.renner.uno's addresses, resolved over DoH at run time
(cidr_complement, in bash; fails above Hetzner's 500 effective rules;
one tcp rule per 100 prefixes, Hetzner's undocumented per-rule limit).
The egress check pod (restore-test/egress-check.yaml) runs at the end of
hcloudtest and again before the restore test writes any secret.
FLUX_SOURCE=restore-test bootstrap.sh flux runs against that cluster only
(its nodes must all be hcloud-*, and KUBECONFIG/TALOSCONFIG must not be the
operator's): flux install, a per-run read-only deploy key on the GitHub
mirror, the restore-test-git Secret, and restore-test/flux-root.yaml on
FLUX_BRANCH. flux-root.yaml applies the production tree; its patches are the
test's whole deviation: the source rewrite, no Ceph cluster network, Velero
ReadOnly, no MariaDB dumps, rotators suspended, Gitea without world egress,
netbird-service-sync at zero replicas and netbird-ingress on no node (the
test cluster rewrote production NetBird's proxy services in run 28), and
backup-verify suspended: it reads the same Hetzner token as hcloudtest and
would delete production's monthly verify VM (both label
managed-by=backup-verify). The patches are labelled 3a to 3h without gaps.
The egress check resolves A and AAAA separately and fails on either error;
before, a failed A lookup went unnoticed and the check ran on IPv6 alone.
Removing a previous egress-check pod fails with a message. hcloudclean checks
for bao like hcloudtest does. site-hcloud.yaml's header says what the site
is for now and that Ceph replicates over the pod network there.
CNPG archives, backs up and prunes as in production: the flux phase writes
cnpg-paths with every archive path in renner-uno-k8s-cnpg-rtest, created
there and emptied and deleted by hcloudclean. A test incarnation with a base
backup in the production bucket would be the newest there, and the next real
recovery would read the test's data. hcloudclean also deletes the test's ssh
keys and the restore-test deploy keys. OpenBao's status wait is bounded at
30 minutes: on a fresh cluster the pod waits for Ceph to provision its
volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.
The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:
- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
<x>-store-source is read by bootstrap.recovery), in a Kustomization of their
own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
substitution blanks every ${...}, which is why it is not on the app
Kustomizations (home-assistant's scripts, the Grafana dashboards). The
ConfigMap is not optional; without it the paths would render empty.
woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
per incarnation (an existing one is kept):
CNPG_<X>_SOURCE the prefix among <x>, <x>-g<14 digits> and the store's
cnpg.renner.uno/source-prefixes whose <x>/base/ holds
the newest completed base backup (backup.info
status=DONE; barman uploads it as STARTED when a
backup begins, barman src/barman/cloud.py
coordinate_backup)
CNPG_<X>_ARCHIVE <x>-g<UTC timestamp>, new and so empty
Every ScheduledBackup has immediate: true, so a new incarnation has a base
backup in its own prefix minutes after it starts, and a later recovery
picks it. S3 access is curl --aws-sigv4 with the restore key (already in
bootstrap.sh's secret list), so the CI image needs no new tool; the
OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
<x>-store-source in externalClusters. serverName stays the cluster name
everywhere. The app Kustomizations depend on cnpg-object-stores, which
depends on infrastructure-cnpg-barman.
Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.
Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
with prune: false (the stack's first PR) when this merges;
cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
not verified; check on one cluster first.
Older generations are not cleaned up: retention applies only to the current
archive prefix.
RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One Velero Restore per Schedule (kubernetes/restore/<schedule>/), applied
by Flux through a restore-<schedule> Kustomization
(clusters/homelab/restore.yaml), restoring the schedule's newest completed
backup. Every app Kustomization with backed-up volumes depends on its own
restore Kustomization, so on a fresh cluster no app starts on an empty
volume, and no app waits for another app's data. This reverses the
decision on #200 (option B, 952e6150): that removed a restore Job that was
never wired in, while the seed-* rsync Kustomizations were still the
bootstrap path. Both are gone; restoring from Velero on a fresh cluster is
now wanted.
Gitea goes first: it is the registry for the in-house images. The gitea
Kustomization depends on restore-gitea alone; every other restore
Kustomization depends on restore-gitea, so Velero moves Gitea's data
first. Each app depends on its own: apps, audiobookshelf (libation through
it), authentik, home-assistant (matter-server, home-assistant-iot, otbr,
whisper too, same namespace and schedule), immich, influxdb, matrix,
music-assistant, navidrome, paperless, trmnl, wordpress. Timeouts follow the
data: immich 6h, audiobookshelf 3h, the rest 1h. One shared gate made
authentik wait for immich's photo library (45 minutes in restore rehearsal
run 28) and held a kustomize-controller worker all that time.
Ordering and failure handling, from Velero release-1.17 source:
- A Restore whose schedule has no synced backup yet goes FailedValidation
for good (restore_controller.go). The restore Kustomizations therefore
depend on velero-backups-synced, which is Ready once the
BackupStorageLocation has completed a sync (status.lastSyncedTime), and
which depends on the velero Kustomization (the BackupStorageLocation and
its credentials).
- After that sync, FailedValidation with "No backups found for schedule"
means there is nothing to restore and counts as done; any other
FailedValidation, Failed or PartiallyFailed fails the Kustomization.
- A claim that already exists is skipped (csi pvc_action: "PVC already
exists"), and the Restore completes with a warning. On the running
cluster the Restores touch nothing.
- Velero deletes a Restore with the backup it came from (30-day TTL);
Flux recreates it, and it again skips the existing claims.
- All StorageClasses bind Immediate, so the data mover does not wait for
a consuming pod.
The velero-schedules header points at the restores instead of the manual
runbook, and bootstrap.sh's wait-apps comment says the volumes come from
Velero. wait-apps waits for gitea, then apps, up to an hour each (the
restore-gitea timeout), since both now include restores.
RUNBOOK-cold-rebuild.md Step 5 describes the restores and how to retry a
failed one: a Restore never retries, and while it is failed every
Kustomization depending on its restore Kustomization stops applying.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five workloads attach a macvlan on ens19 (v30iot) with DHCP IPAM:
shelly-manager and the mDNS relay (inside the home-assistant Kustomization),
matter-server, music-assistant and otbr. On a node without that VLAN the CNI
add fails and the pod never starts.
- k8s-1..3 carry net.renner.uno/iot-vlan: "true"; each of them declares
ens19 in its VLANConfig. The five Deployments select on it. The comments
they replace described the retired k8s-edge taint.
- shelly-manager and the mDNS relay move to apps/home-assistant-iot, with a
Kustomization of their own (depends on gitea, home-assistant and the
infrastructure Kustomizations). home-assistant's wait: true no longer
covers them, so HA and everything waiting on it (evcc, trmnl,
zone-configurator, speech-to-phrase) are Ready without the VLAN, and a
broken IoT bridge no longer holds them back in production either.
- shelly-manager-auth stays in apps/home-assistant: mqtt-account-controller
mounts it too, and that Kustomization comes first. Moved with
shelly-manager, mqtt-account-controller would wait for a Secret that only
the later Kustomization creates.
- The probes-blackbox and authentik blueprint comments name the new paths.
On a site without the label the IoT Kustomizations fail at their timeout,
naming the Pending pods; nothing else depends on them.
Production rollout:
1. Label the nodes first (talosctl patch the nodeLabels), or the five pods go
Pending.
2. home-assistant must run with prune: false (the stack's first PR) when
this merges, so it drops shelly-manager and the mDNS relay from its
inventory without deleting them and home-assistant-iot adopts them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llama-server, llama-embed, wiki-embed (gpu.intel.com/i915) and the wiki
behind them are bound to the Intel iGPU. On a site without one (Hetzner,
restore rehearsal runs 27/28) their pods stayed Pending, and the GPU plugin
and exporter DaemonSets sat on every node with nothing to serve.
The node patches k8s-1..3 label their machines
hardware.renner.uno/intel-gpu=true. The GPU plugin, the exporter, the
three llama Deployments and the wiki now select on it. Elsewhere the
DaemonSets have no pods and the Deployments stay unscheduled.
Production rollout order: the label must be on the nodes (talosctl patch,
no reboot) before these manifests reconcile, or the GPU plugin and the
llama/wiki pods are removed or left Pending.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
crowdsec, external-dns-unifi, netbird-ingress and velero were in the
infrastructure Kustomization, while the ExternalSecrets that create their
Secrets were in infrastructure-crds, which depends on infrastructure. On a
cold start each was therefore installed before its Secret could exist: the
pods sat in CreateContainerConfigError until OpenBao was unsealed and ESO
synced. For the HelmReleases (crowdsec, external-dns-unifi, velero) Helm's
wait timed out meanwhile; with remediation.retries: 3 they give up after
four attempts and stay failed even once the Secret appears. Velero failing
that way stops every restore.
| workload | Secrets |
|---|---|
| crowdsec | crowdsec-bouncer-key (crowdsec, ingress), crowdsec-edge-bouncer-key |
| external-dns-unifi | external-dns-unifi-secret |
| netbird-ingress | netbird-ingress-setup-key |
| velero (+ backup-verify CronJob) | cloud-credentials, cloud-credentials-hetzner, hcloud-backup-verify |
Each now has a Kustomization of its own (clusters/homelab/
infrastructure-services.yaml), depending on infrastructure and
infrastructure-crds, with its ExternalSecrets beside it. The openbao
ClusterSecretStore in infrastructure-crds turns Ready only once OpenBao is
unsealed, so the Secret exists before the consumer installs. One
Kustomization each so that a broken service holds back only itself.
Velero's Schedules need the velero.io CRDs from its HelmRelease, so they move
from post-crds to infrastructure/velero-schedules with a Kustomization after
velero.
post-crds keeps cloudflare-api-token (consumed by its own ClusterIssuer) and
flux-git. The directories themselves stay where they are, so paths such as
the CI test infrastructure/velero/backup-verify/tests/run.sh are unchanged.
netbird-ingress gets 10m to roll its DaemonSet over three nodes
(maxUnavailable 1, minReadySeconds 45). The crowdsec comments name the
Kustomization it now runs in, and flux-graph.sh colours the new nodes.
Production rollout: infrastructure and infrastructure-crds must run with
prune: false (the stack's first PR) when this merges. Pruned, the velero
HelmRelease would be uninstalled, and the moved ExternalSecrets would take
their Secrets with them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bird and netbird-service-sync run images from Gitea's registry with no
copy on ghcr.io. In the infrastructure Kustomization they started long
before the restored Gitea served them and sat in ImagePullBackOff (restore
rehearsal runs 27, 28). infrastructure cannot depend on gitea, which
depends on it, so both move to a new Kustomization, infrastructure-inhouse,
that depends on infrastructure, infrastructure-crds and gitea. Their
namespace stays in infrastructure. e2e-test.sh checks the new Kustomization
for Ready.
Images published to ghcr.io too have a fallback in the nodes' registry
mirror and need no ordering; multus cni-plugins stays in infrastructure.
The secret-rotator and backup-verify CronJobs also use Gitea images but
only run on schedule; they are left in place.
bird fills routing table 47, which exists only where the site patch routes
pod traffic into it, and renders its config from node annotations only that
site sets. On a site without it the template rendered <no value> and the
init container crash-looped (run 28). The DaemonSet now selects on
net.renner.uno/pod-table=47, which site-homelab.yaml puts on its nodes.
Production rollout:
1. The pod-table label must be on the nodes (talosctl patch, no reboot)
before the DaemonSet change reconciles, or bird is removed from every
node and pod traffic falls through to the main table.
2. infrastructure must run with prune: false (the stack's first PR) when
this merges. It then drops bird and netbird-service-sync from its
inventory without deleting them, and infrastructure-inhouse adopts
them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reads every kustomize.toolkit.fluxcd.io/v1 Kustomization in
kubernetes/clusters/homelab and emits Graphviz DOT, one edge per dependsOn
entry, pointing from the dependency to its dependent. Each node carries its
wait mode and timeout, drawn as Flux actually checks it: a Kustomization
with healthChecks and no wait is health-checked (as infrastructure is), one
with neither has no check at all, whatever its healthCheckExprs say.
Infrastructure is green, the Velero restore chain blue, wait: false dashed,
suspended Kustomizations grey.
With no argument or a .dot path it writes the full graph; with a .svg or .png
path it renders the transitive reduction (tred). Generated rather than
checked in, so it cannot drift from the manifests. Works with either yq
(mikefarah's or the Python wrapper): yq only converts YAML to JSON, jq builds
the DOT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The authentik chart renders server/servicemonitor.yaml whenever
server.metrics.serviceMonitor.enabled is set, without checking
.Capabilities for monitoring.coreos.com (verified in chart 2026.8.3). The
authentik Kustomization does not depend on monitoring, so on an empty
cluster the Helm install fails until kube-prometheus-stack has installed the
CRDs. The chart's ServiceMonitor is switched off and declared standalone in
monitoring-rules, as cilium and velero already do. The selector matches the
live Service authentik-server-metrics (port metrics, 9300); the new name
differs from the chart's authentik-server, so the two never contend for one
object while production rolls over.
authentik-outpost-servicemonitors.yaml hand-wrote a headless Service
ak-outpost-local-radius-metrics and a ServiceMonitor for it, because the
radius outpost controller used not to expose its metrics port. authentik
2026.8.3 does: the operator creates that Service (http-metrics 9300) and its
own ServiceMonitor. On a cold cluster the operator created the Service first,
with a ClusterIP; clusterIP is immutable, so the dry-run of the hand-written
clusterIP: None copy failed and took the whole monitoring-rules
Kustomization with it (restore test run 29). The file goes; the operator
owns both. No alert rule refers to the radius job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a cold cluster (restore rehearsal runs 22-28) apps started before the
things they need and sat in ImagePullBackOff or CreateContainerConfigError,
while their Kustomizations reported Ready or held kustomize-controller
workers. The dependency graph now states those needs:
- infrastructure health-checks the openbao StatefulSet, not only the
HelmRelease (Ready as soon as the chart is installed, sealed or not). Its
readiness probe is `bao status`, which fails while sealed, so 3/3 ready
means every replica is unsealed; infrastructure-crds no longer applies the
ClusterSecretStore against a sealed OpenBao, which External Secrets then
backs off for minutes.
- gitea edges: apps whose images live only in Gitea's registry (authentik,
home-assistant, paperless, review-bot, wiki) and apps that are front ends
to Gitea (gitea-mcp, woodpecker, renovate). Images also published to
ghcr.io have a fallback in the nodes' registry mirror and get no edge
(apps' ical-proxy, trmnl's alpine-jq).
- monitoring depends on influxdb: Grafana mounts grafana-influxdb-token,
which a Job in the influxdb app writes.
- evcc depends on home-assistant: mqtt-account-controller there delivers
mqtt-evcc-client-tls, and mosquitto is its broker.
- trmnl depends on home-assistant (the renderer reads HA's API);
home-assistant depends on influxdb (its InfluxDB integration writes there).
- home-assistant and wordpress wait for their workloads. With wait: false
and no healthChecks, Ready means only that the manifests applied, and
healthCheckExprs are never evaluated (kustomize-controller's checkHealth
returns first). otbr is suspended: it is not in use.
- home-assistant's timeout is 30m, not 4h. The 4h was sized for a seed
initContainer that waited up to 3h for an HAOS backup; the slowest part
left is ha-pg recovering from barman. With wait: true the 4h would also
hold a kustomize-controller worker that long on a failed cold start.
No cycles: gitea depends only on the infrastructure layers; influxdb does
not use monitoring's CRDs; home-assistant does not reach evcc or trmnl.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The k8s-gateway values were read with `yq -y`, a flag only the Python yq
has. The vmtest image installs Alpine's `yq`, which is yq-go (mikefarah),
so the talos phase would fail there. Without the flag both print a
document helm accepts, as the Cilium values line already relies on.
- A cni-confdir pod left from an earlier run (Failed or Succeeded) made
`kubectl apply` a no-op and the wait time out. The step deletes it first.
- The Cilium HelmRelease comment named itself the only place Cilium is
configured; the site patch's cilium-site ConfigMap is the other.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What differs between sites moves out of cluster-wide config into the Talos
site patches (site-homelab.yaml, site-qemu.yaml), and the consumers read it
from there:
- nodeIP.validSubnets lists IPv4 first, as the live nodes run. Talos keeps
the order (siderolabs/net FilterIPs) and the first entry becomes the
primary node IP; with fd02:46::/64 first, kube-apiserver exits with
'service IP family "10.96.0.0/12" must match public address family'.
- etcd.advertisedSubnets (VLAN 46, a fact of the home network) moves
unchanged from cluster.yaml to site-homelab.yaml; the rendered production
config is byte-identical.
- Pod routing (routingMode, autoDirectNodeRoutes, directRoutingDevice) and
the native-routing CIDRs are a cilium-site ConfigMap per site patch, read
by the Cilium HelmRelease via valuesFrom. bootstrap.sh merges it the same
way: site values first, HelmRelease values on top. Production renders
unchanged (helm template, TLS secrets excluded).
- Each site patch states its node resolver list (ResolverConfig: k8s-gateway
first, then the site's resolver) and k8s-gateway's upstream as a
k8s-gateway-site ConfigMap, read by the k8s-gateway HelmRelease via
valuesFrom. bootstrap.sh installs k8s-gateway right after the nodes are
Ready, from the HelmRelease under the site values, and Flux adopts it:
until k8s-gateway answers, every lookup waits out the node's 2 s read
timeout and pods resolve nothing, Flux included.
- bootstrap.sh creates Cilium's cni.confPath subdirectory with a
hostNetwork pod per node (/etc/cni is read-only to the kubelet on Talos)
and applies Multus right after Cilium; without it the kubelet reports
"cni plugin not initialized" and nothing schedules on a cold cluster.
- site-homelab.yaml labels its nodes net.renner.uno/pod-table=47 (table 47
exists only where the site routes pod traffic into it); k8s-1..3 label
hardware.renner.uno/intel-gpu and net.renner.uno/iot-vlan, the hardware
each machine has. The workloads select on these labels in later changes.
Rollout order for production: the cilium-site and k8s-gateway-site
ConfigMaps must exist before the HelmReleases reconcile (valuesFrom is not
optional). Talos creates inline manifests on a config change but never
updates them; changes go through talosctl upgrade-k8s. The node labels go
on with talosctl patch, no reboot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
serviceaccounts.yaml and the rook-ceph operator HelmRelease still said the
ceph-csi-drivers chart is not installed and the SAs and RBAC are only
rendered statically.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The controller mounted mqtt-home-assistant-tls as a required volume, and is
the only creator of that Secret (it applies the cert-manager Certificates of
the register's cert accounts). On a new cluster - Velero restores volumes,
not Secrets - its pod could never start, and mosquitto, Home Assistant,
Zigbee2MQTT and evcc stayed in ContainerCreating behind the Secrets it would
have created.
Optional now. Verified in konairius/mqtt-account-controller main: the cert
dir is read only by the HA cert sync (02:00-05:00 window, errors logged);
the broker is dialled lazily; Secret.Read treats a missing Secret as empty;
Reconcile runs reconcileCerts even when the broker step fails. So the
controller starts, creates the Certificates, the consumers start, and the
next loop reconciles the broker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The claim named its production PV, which does not exist on a rebuilt
cluster: the claim stayed Pending there, and a restored claim bound to a
new volume made the gitea Kustomization fail on the immutable field.
volumeName leaves git and the gitea Kustomization ignores
/spec/volumeName on this claim. kustomize-controller is the field's
only manager on the live claim, so its ignore rule adopts the live value
instead of clearing it; size growth still applies.
To verify after merge: the gitea Kustomization reconciles Ready and the
live claim still names pvc-e7c56db2-481d-40cb-b94a-b264b7b4b5be.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since Rook 1.20 the CSI Driver and OperatorConfig resources come from the
ceph-csi-drivers chart (rook Documentation/Helm-Charts/csi-drivers-chart.md:
"Rook requires the installation of the Ceph-CSI Driver chart"). The chart
was never installed here: the live cluster's Drivers are the ones the
operator migrated once during the 1.19->1.20 upgrade (2026-05-09), and
PR #12 only rendered the ServiceAccounts and RBAC. A fresh cluster
therefore has no CSI driver and provisions no volume: found by the
Hetzner restore test, where every PVC stayed Pending ("Waiting for a
volume to be created either by the external provisioner
'rook-ceph.rbd.csi.ceph.com'").
Values: Rook's recommended values for 1.21.0 plus the live shape:
2 controller replicas with Recreate, the workload-class=pinned
toleration on the node plugins, a snapshotter for RBD (Velero's CSI
snapshots of ceph-block) and group snapshots for CephFS.
The chart's ServiceAccounts are named per driver
(rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa, ...), so they do not collide
with the static ones; the static ServiceAccounts and RBAC stay until the
live CSI pods run on the chart's, and go in a second change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restore-test stack moves objects between Kustomizations:
- bird and netbird-service-sync to infrastructure-inhouse;
- crowdsec, external-dns-unifi, netbird-ingress, velero and the Velero
Schedules, plus their ExternalSecrets, to their own Kustomizations;
- shelly-manager and the mDNS relay to home-assistant-iot;
- the CNPG ObjectStores to cnpg-object-stores.
Flux's garbage collection deletes an object dropped from a Kustomization's
source if its live owner labels (kustomize.toolkit.fluxcd.io/name and
/namespace) still name that Kustomization. Those labels belong to whichever
Kustomization applied the object last. Two phases (add to the new owner,
then remove from the old one) therefore do not guarantee anything: if the
old owner applied last before phase 2, it prunes the object
(kustomize-controller v1.9.6 deleteObjects, fluxcd/pkg/ssa v0.76.2
manager_delete.go).
With prune: false the giving Kustomizations drop the moved objects from
their inventory without deleting them, and the new owners adopt them. This
covers infrastructure, infrastructure-crds, home-assistant, authentik,
immich, matrix, paperless, trmnl and woodpecker. The last PR of the stack
switches prune back on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Mistral, Anthropic and TypeSafe keys come from the bot's own OpenBao path
apps/review-bot (minstral-api-key, antropic-api-key, typesafe-api-key) instead
of the wiki's keys. All keys and both tokens are in review-bot-secrets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mistral cut every POST /conversations off with HTTP 502 exactly ten minutes
after it started (about 30 times in the last 7 days), and every retry restarted
the whole review, so a review needing more than ten minutes could never finish.
- Mistral: the conversation now streams (stream=true, SSE). A stream that
breaks off after the conversation started is looked up in the stored history
(store=true, GET /conversations/{id}/history) before it is asked again. A
broken stream is retryable like 429/502/503/504; a
conversation.response.error event is not. One agent per model, named
<AGENT_NAME>-<model>.
- Anthropic: new backend on the Messages API (Go SDK, streaming, adaptive
thinking, effort high). Anthropic connects to the Gitea MCP server
(gitea-mcp.renner.uno/mcp) with a read-only token; the toolset is in
allowlist mode with only the read tools enabled. Paused turns are resumed,
refusals use the server-side fallback on Opus/Sonnet 5.5.
- Models: MODELS lists backend:model = description, the first is the default.
A backend whose keys are missing is switched off with a warning instead of
stopping the bot.
- Selector: SELECTOR=typesafe asks TypeSafe (Jev, Choice) which model reviews
each PR, from the title, description, size and changed files. A failure or
an unusable choice falls back to the default model.
- Secrets: the Anthropic and TypeSafe keys reuse the wiki keys
(apps/ai/wiki-anthropic, apps/ai/wiki-typesafe), like the Mistral key does.
The MCP token is a second review-bot PAT scoped read:repository,read:issue,
read:user, minted by a new rotate-mint-mcp step into
infrastructure/review-bot#mcp_token (seeded with a placeholder). Each
optional key has its own ExternalSecret and optional env ref.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wp-pg-store-source, the recovery source of wp-pg's bootstrap.recovery,
pointed at s3://renner-uno-k8s-cnpg/wp-pg: the prefix of the retired
apps/wp-pg. Since the namespace move wp-pg archives to wp-pg-woodpecker
(wp-pg-store). A cold rebuild would therefore have restored woodpecker as of
2026-08-18 and reported success.
Bucket, listed read-only 2026-10-16:
- wp-pg/wp-pg/base: newest 20260818T023502
- wp-pg-woodpecker/wp-pg/base: 20260916T024000 .. 20261016T024000 (daily),
wals/ through timeline 6 (00000006.history from 2026-09-23)
The source now reads wp-pg-woodpecker; serverName stays wp-pg, the name the
live cluster archives under. bootstrap.recovery is honoured only when PGDATA
is absent, so the running cluster is unaffected. The old prefix wp-pg is no
longer referenced and keeps its objects; nothing applies retention to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ingress routes only the /status prefix; /hook and /metrics stay
cluster-internal. No public label, so it is not on the NetBird proxy.
The new CNP lets Traefik reach the pod on 8080.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every phase of a pull request's review is now visible in two places:
- a commit status "review-bot" on the reviewed head: pending while
queued, running or waiting to retry; success once reviewed (with the
verdict) or skipped; error with the reason when the review failed;
warning when new commits superseded the reviewed head. The statuses
are posted one at a time in order, so a later phase never lands before
an earlier one, and never block the queue.
- GET /status and GET /status/{owner}/{repo}/{n}: the in-memory state
of every PR seen since start, with the queue position while queued.
Finished entries are dropped after a day.
Until now a failed review posted nothing, so a PR whose review failed
looked the same as one still waiting.
Also gofmt report_test.go, which the CI gofmt check flags on master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
containerd pulls in-house images from gitea.renner.uno in the host
network namespace, through Talos host DNS. Its only upstream was the
UDM (from DHCP), so every node-level pull of an in-cluster registry
depended on an external DNS.
A ResolverConfig document lists k8s-gateway (10.96.0.11) first and the
UDM second. It replaces the DHCP nameservers. Talos host DNS tries
upstreams in order and moves on after an error or SERVFAIL/REFUSED
(internal/pkg/dns/dns.go); a silent upstream costs the 2 s proxy read
timeout (coredns plugin/pkg/proxy), inside the 4.5 s per-query
deadline, so a cold node still resolves through the UDM until Cilium
and k8s-gateway are up.
Dry run on k8s-1: applies without reboot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes the Paperclip deployment (namespace paperclip, CNPG cluster
pclp-pg, ingresses, image automation, server image build) and the
infrastructure that existed only for its agents:
- agent sandbox: paperclip-renner namespace and CNPs, the
kubernetes-sigs/agent-sandbox controller with its namespace, CNP and
Renovate rules
- agent-runtime-{go,ops} images and the build-agent-runtime pipeline
- matrix bridge (Element <-> Paperclip agents) and synapse's appservice
registration mount
- monitoring loki-read-gateway (only called by the paperclip Role)
- MQTT account agents-read and the controller's RBAC in ns paperclip
- the OpenBao secret-exists-check policy file
- agent admits in the wiki, lidarr, shelly-manager, mosquitto and
hubble-relay CNPs; the paperclip-api Gitea webhook host; the wiki API
key; the ClusterSecretStore namespace entry; the pclp-pg RECOVERY row
- the Paperclip design spec and plan
Authentik group "Paperclip Users" is switched to state: absent so the
blueprint deletes it. package-retention's EXCLUDE_PACKAGES becomes
optional, since its only value was the agent-runtime prefix.
The shelly-manager api-proxy sidecar stays: mqtt-account-controller
calls shelly-manager-api too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chart 4.0.0 (installed 2026-10-08 via the open semver range) replaced
domain/watchedResources/fallthrough/extraZonePlugins with a servers:
list. The old keys were ignored and the chart default ran instead:
k8s_gateway example.com watching Service/HTTPRoute, forward to
/etc/resolv.conf, Service type LoadBalancer. No renner.uno Ingress name
has been answered in-cluster since.
The new values render the same Corefile as the last 3.x release
(helm rev 7): .:1053, k8s_gateway renner.uno on Ingress with
fallthrough renner.uno., cache with serve_stale, forward to the UDM,
Service ClusterIP 10.96.0.11 port 53, runAsUser 1000. Verified with
helm template against chart 4.0.1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 3 of unifying the auth flows. After #1285/#1286 no provider, brand
or flow references the passkey-only objects any more (checked live:
0 providers on either flow; passkey-webauthn-validation and
passkey-passwordless-identification are bound only inside the two
retired flows). They are now declared absent: both flows (their stage
bindings and the policy bindings on those cascade), the
passkey-passwordless-identification stage, the passkey-webauthn-validation
stage and the passkey-user-has-webauthn policy. The shared default
stages they bound (password, MFA validation, login) stay.
The passkey-stepup.yaml key keeps its name, so the blueprint instance
stays the same; it now only carries the OAuth2 providers and the
removals.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k.renner
claude-botandClaude Opus 5.5
security-bot