crowdsec, external-dns-unifi, netbird-ingress and velero were in the
infrastructure Kustomization, while the ExternalSecrets that create their
Secrets were in infrastructure-crds, which depends on infrastructure. On a
cold start each was therefore installed before its Secret could exist: the
pods sat in CreateContainerConfigError until OpenBao was unsealed and ESO
synced. For the HelmReleases (crowdsec, external-dns-unifi, velero) Helm's
wait timed out meanwhile; with remediation.retries: 3 they give up after
four attempts and stay failed even once the Secret appears. Velero failing
that way stops every restore.
| workload | Secrets |
|---|---|
| crowdsec | crowdsec-bouncer-key (crowdsec, ingress), crowdsec-edge-bouncer-key |
| external-dns-unifi | external-dns-unifi-secret |
| netbird-ingress | netbird-ingress-setup-key |
| velero (+ backup-verify CronJob) | cloud-credentials, cloud-credentials-hetzner, hcloud-backup-verify |
Each now has a Kustomization of its own (clusters/homelab/
infrastructure-services.yaml), depending on infrastructure and
infrastructure-crds, with its ExternalSecrets beside it. The openbao
ClusterSecretStore in infrastructure-crds turns Ready only once OpenBao is
unsealed, so the Secret exists before the consumer installs. One
Kustomization each so that a broken service holds back only itself.
Velero's Schedules need the velero.io CRDs from its HelmRelease, so they move
from post-crds to infrastructure/velero-schedules with a Kustomization after
velero.
post-crds keeps cloudflare-api-token (consumed by its own ClusterIssuer) and
flux-git. The directories themselves stay where they are, so paths such as
the CI test infrastructure/velero/backup-verify/tests/run.sh are unchanged.
netbird-ingress gets 10m to roll its DaemonSet over three nodes
(maxUnavailable 1, minReadySeconds 45). The crowdsec comments name the
Kustomization it now runs in, and flux-graph.sh colours the new nodes.
Production rollout: infrastructure and infrastructure-crds must run with
prune: false (the stack's first PR) when this merges. Pruned, the velero
HelmRelease would be uninstalled, and the moved ExternalSecrets would take
their Secrets with them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bird and netbird-service-sync run images from Gitea's registry with no
copy on ghcr.io. In the infrastructure Kustomization they started long
before the restored Gitea served them and sat in ImagePullBackOff (restore
rehearsal runs 27, 28). infrastructure cannot depend on gitea, which
depends on it, so both move to a new Kustomization, infrastructure-inhouse,
that depends on infrastructure, infrastructure-crds and gitea. Their
namespace stays in infrastructure. e2e-test.sh checks the new Kustomization
for Ready.
Images published to ghcr.io too have a fallback in the nodes' registry
mirror and need no ordering; multus cni-plugins stays in infrastructure.
The secret-rotator and backup-verify CronJobs also use Gitea images but
only run on schedule; they are left in place.
bird fills routing table 47, which exists only where the site patch routes
pod traffic into it, and renders its config from node annotations only that
site sets. On a site without it the template rendered <no value> and the
init container crash-looped (run 28). The DaemonSet now selects on
net.renner.uno/pod-table=47, which site-homelab.yaml puts on its nodes.
Production rollout:
1. The pod-table label must be on the nodes (talosctl patch, no reboot)
before the DaemonSet change reconciles, or bird is removed from every
node and pod traffic falls through to the main table.
2. infrastructure must run with prune: false (the stack's first PR) when
this merges. It then drops bird and netbird-service-sync from its
inventory without deleting them, and infrastructure-inhouse adopts
them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes the Paperclip deployment (namespace paperclip, CNPG cluster
pclp-pg, ingresses, image automation, server image build) and the
infrastructure that existed only for its agents:
- agent sandbox: paperclip-renner namespace and CNPs, the
kubernetes-sigs/agent-sandbox controller with its namespace, CNP and
Renovate rules
- agent-runtime-{go,ops} images and the build-agent-runtime pipeline
- matrix bridge (Element <-> Paperclip agents) and synapse's appservice
registration mount
- monitoring loki-read-gateway (only called by the paperclip Role)
- MQTT account agents-read and the controller's RBAC in ns paperclip
- the OpenBao secret-exists-check policy file
- agent admits in the wiki, lidarr, shelly-manager, mosquitto and
hubble-relay CNPs; the paperclip-api Gitea webhook host; the wiki API
key; the ClusterSecretStore namespace entry; the pclp-pg RECOVERY row
- the Paperclip design spec and plan
Authentik group "Paperclip Users" is switched to state: absent so the
blueprint deletes it. package-retention's EXCLUDE_PACKAGES becomes
optional, since its only value was the agent-runtime prefix.
The shelly-manager api-proxy sidecar stays: mqtt-account-controller
calls shelly-manager-api too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- infrastructure/agent-sandbox: Flux GitRepository (tag v1.0.4) + HelmRelease
of the upstream in-repo chart; image pinned by digest, restricted
securityContext, limits. CRDs via crds: CreateReplace.
- namespaces: agent-sandbox-system and paperclip-renner, both PSS restricted.
- network-policies: CNP for agent-sandbox-system (probes in, apiserver out).
- apps/paperclip/sandbox-rbac.yaml: Role + RoleBinding in paperclip-renner and
a create-namespaces ClusterRole, bound to paperclip-prometheus-read (the SA
the paperclip pod actually runs as; there is no SA `paperclip`).
- renovate.json: group the git tag and the image pin into one PR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- kubernetes/infrastructure/netbird-proxy-services -> netbird-service-sync:
SA, ClusterRole(Binding) and Deployment renamed, in namespace
netbird-service-sync, image gitea_admin/netbird-service-sync pinned to
the master build digest, BAO_ROLE=netbird-service-sync.
- Old policy doc netbird-proxy-services.hcl removed; Traefik comment
follows the name.
The infrastructure KS owns both old and new objects, so prune and apply
happen in one pass. OpenBao policy and role netbird-service-sync are
already written.
Closes#876
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing reads table 48 yet. The mesh fills it so the routing rules land on a table that is already correct, and so a mistake here shows as an empty table rather than as broken pod traffic.
No address is written down twice. Each node publishes its pod0 addresses as Talos nodeAnnotations, beside the VLANConfig that assigns them; the init container reads the node list and renders from it. bird.conf holds no address at all, and the pod CIDRs are not restated either — the own-CIDR filter keys on the outgoing interface plus a maximum-length exclusion, which separates the node's pod route from cilium_host's own host address without naming a prefix.
Rendering is go-template-file, not string building: kubectl renders the two templates, a field selector drops the node itself server-side, and in-cluster config removes the token and CA plumbing. kubectl comes from the Alpine package, so there is no version literal to drift out of sync and no download to trust; it only ever reads core/v1 Nodes. jq and curl are gone.
The init container ends with bird -p, so BIRD's own parser gates startup: a node whose annotation is missing renders as <no value> and fails there rather than crashlooping the daemon or forming a mesh with a hole.
learn all rather than learn, since plain learn skips proto kernel. direct is explicit because iBGP defaults to multihop, which would make the gateway recursive. strict bind keeps the listener off mgmt0 and ingress0.
Verified against the live cluster with all three nodes annotated via --mode=try: the templates rendered the expected defines and four peer protocols, and bird -p parsed the result inside the image with exit 0 — which also confirms net.len and the include ordering. kubectl kustomize builds.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deployment for the controller of the previous commit: openbao namespace
like the rotators (its default-deny already allows OpenBao, DNS, the
apiserver and world), own ServiceAccount with a read-only ClusterRole
(Ingresses list/watch, Services get), one replica, Recreate. The image tag
is the branch build that pushed this exact code.
The OpenBao role is applied once by hand (policies/netbird-proxy-services
.hcl); until then the pod logs an auth error per minute and changes nothing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Backing up an external server is that server's own business. The pull path
made the cluster responsible for it and, worse, put a root key on the VPS
that the cluster held — a standing inbound grant that existed only to move
a SQLite file.
Removed cluster-side: the netbird-backup namespace and everything in it
(the nightly CronJob, the netbird-backup-ssh ExternalSecret, the 5Gi
ceph-block PVC), the namespace's entry in the OpenBao ClusterSecretStore
allow-list, the netbird-backup-daily Velero Schedule, and the
NetbirdVpsBackupStale / NetbirdVpsBackupNeverRan alerts.
Flux prunes the PVC, which destroys the 14 retained dumps (2026-08-15
through 08-28). That is intended. Velero's existing offsite copies age out
on their own under the schedule's 720h TTL; no cleanup is needed there.
Removed VPS-side: the dump script, and the forced-command key authorising
it. Deleting the two tasks that placed them would only stop ansible
managing them and leave both live on the host, so they are replaced with
state: absent tasks that actually revoke them. Those two can go once the
role has run against every host that ever carried them.
The trmnl-daily schedule's stagger comment pointed at netbird-backup-daily
for its slot; it now points at apps-daily, which is the schedule that
actually precedes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the Intel GPU device plugin as a DaemonSet in kube-system, making the
on-board Iris Xe of each MS-01 node schedulable as `gpu.intel.com/i915`.
Pods can then request the iGPU without a /dev/dri hostPath or privilege
escalation — the plugin passes the render node through the kubelet
device-plugin API. The resource covers QuickSync transcoding as well as GPU
compute.
Vendored from upstream v0.36.0 rather than installed from the
`intel-device-plugins-gpu` Helm chart: that chart renders a `GpuDevicePlugin`
custom resource requiring the Intel Device Plugin Operator, its CRDs and a
cert-manager webhook. One device type on three identical nodes gives the
operator nothing to manage that the DaemonSet does not.
Runs in kube-system because the plugin mounts hostPaths; the cluster enforces
PodSecurity `baseline`, which forbids those, and kube-system is the only
namespace in the API server's exemption list. NFD is not deployed, so
upstream's nfd_labeled_nodes overlay and its nodeSelector are omitted.
`-shared-dev-num=4` lets several containers share the single iGPU.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
WordPress speaks MySQL natively. The PostgreSQL route would mean pg4wp, an
SQL-rewriting shim whose README lists WordPress 6.5.3 as its newest tested
version while renner.uno runs 7.0.4 — every core update would become a
compatibility gamble.
Charts are OCI artifacts, so this follows the OCIRepository pattern from
spegel rather than the HelmRepository pattern used for CNPG. CRDs ship as a
separate chart that upstream requires first, hence the dependsOn. The version
range is capped at the year boundary: the project uses calendar versioning and
ships an upgrade guide with each minor.
renner-uno-k8s-mariadb carries the same 30d noncurrent-version expiry as the
CNPG and Velero buckets, matched to the Backup resource's maxRetention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Closes the last big SPOF-assessment item: the management SQLite store
(peers, setup keys, groups, routes) had zero backup coverage — losing
the Hetzner VPS meant re-enrolling the entire mesh. The VPS cannot
reach any PBS (both PBS names resolve to the public proxy, ports
closed), so the cluster pulls instead of the VPS pushing:
- VPS (netbird-server role): a dump script takes a CONSISTENT snapshot
via sqlite3 .backup (online-backup API, safe against the live writer)
plus /opt/netbird/config and streams a tar.gz to stdout. A dedicated
ed25519 key is authorized for root restricted to exactly that script
(restrict,command=…) — the puller can run nothing else.
- Cluster (kubernetes/infrastructure/netbird-backup): nightly CronJob
(02:45) ssh-pulls the dump onto a 5Gi Ceph PVC, gzip-verifies before
renaming, keeps 14 dumps; ExternalSecret syncs the private key from
OpenBao (homelab/infrastructure/netbird-backup, one-time manual
provisioning like the rotator bootstraps); restricted-profile pod.
- Offsite: new Velero Schedule netbird-backup-daily (07:45, PVC-only,
snapshotMoveData, 30 d TTL) ships the PVC to Hetzner Object Storage.
- Alerts: NetbirdVpsBackupStale (critical, >26 h without success) and
NetbirdVpsBackupNeverRan (warning — catches the unprovisioned-secret
state so the backup cannot die silently).
Restore: untar over /opt/netbird/{management,config} after the role
rebuilds the stack.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
End-to-end cluster-internal rotation of the shared server<->agent gRPC secret, no per-rotation human step:
- rotate-agent-secret.yaml: quarterly CronJob (+ on-demand) logs into OpenBao via k8s-auth role 'rotator', generates a random value, bao kv patch's homelab/apps/woodpecker (Gitea OAuth fields untouched).
- infrastructure/reloader: Stakater Reloader restarts annotated workloads on Secret change.
- server.yaml/agent.yaml: reloader annotation for woodpecker-secret.
- externalsecret.yaml: refreshInterval 1h -> 10m so a rotation propagates within ~10m.
- openbao/policies/rotator.hcl: least-privilege policy (patch on the one path only); NOT Flux-applied, one-time manual bootstrap documented in-file (eso-read.hcl convention).
PoC scope: woodpecker only. Requires a one-time OpenBao bootstrap (rotator policy + role) before the CronJob succeeds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The woodpecker-ci namespace enforces pod-security `privileged` (required for
the allow-listed docker/buildx plugin), Woodpecker runs WOODPECKER_OPEN=true,
and no admission controller exists. PodSecurity is all-or-nothing per
namespace, so the buildx exemption also un-gates hostPath/hostNetwork/hostPID/
hostIPC and privileged on every step pod a pipeline author can shape.
Reduce the blast radius with three defense-in-depth controls, none of which
weaken the actual build capability (this repo's builds use rootless kaniko;
only the external buildx plugin needs privileged, and it stays allow-listed):
- ValidatingAdmissionPolicy (GA/in-tree on k8s v1.35, no new controller):
denies host namespaces + hostPath for all step pods, and privileged for
every image except woodpeckerci/plugin-docker-buildx. Re-narrows the
PodSecurity exemption at the authoritative K8s admission layer. CEL verified
to compile via server-side dry-run.
- CiliumNetworkPolicy: ingress default-deny for step pods (house style,
egress left default-allow so builds keep reaching the internet + Gitea).
- ResourceQuota + LimitRange: bound pods/CPU/memory/PVCs against a
resource-exhaustion DoS; sized generously so current builds are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OpenBao was initialized and configured today (kv-v2 at homelab/, k8s
auth, eso-read policy + role bound to the external-secrets SA); the
store can therefore turn Ready and no longer endangers the post-crds
wait gate. Audit device moves into the server HCL because OpenBao 2.5
rejects API-managed audit devices; log_raw stays off so Loki only ever
sees HMAC-redacted values.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
Single-node raft OpenBao (5Gi ceph-block, UI at openbao.k.renner.uno,
internal-only ingress) and ESO 2.x. No ClusterSecretStore yet — it can
only turn Ready after manual init/unseal, and committing it now would
freeze the infrastructure-crds wait gate (2026-07-13 incident shape).
Full phased plan incl. per-file checklist and DR runbook in
kubernetes/MIGRATION-sops-openbao.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
All 8 public hostnames have moved off the tunnel to the NetBird reverse
proxy, so cloudflared routes nothing but a 404. Remove the component
entirely:
- delete kubernetes/infrastructure/cloudflared/ (Deployment, ConfigMap,
tunnel-credentials Secret, Kustomization) + drop it from
infrastructure/kustomization.yaml (Flux prunes the workload)
- remove the cloudflared image-automation policy + README row
- drop the cloudflared-to-ingress-only CiliumNetworkPolicy
- scrub stale "cloudflared" comments (netbird-ingress, k8s-gateway,
ingress-default-deny external-entry note)
Cloudflare is still used for DNS (external-dns-cloudflare) and
cert-manager DNS-01 — only the tunnel is retired. The Cloudflare-side
tunnel itself (Zero Trust) can be deleted separately.
MERGE LAST — only after every per-service proxy cutover is live and
verified; cloudflared already serves no routes, so this just removes the
dormant workload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a second, parallel ingress path over the NetBird overlay mesh
(100.96.0.0/16), in addition to the existing Cloudflare path. A new
in-cluster NetBird routing peer enrolls with the self-hosted management
server (netbird.renner.uno) and advertises the Traefik VIP (10.2.99.86)
into the mesh, so every Traefik-fronted service becomes reachable from
mesh peers with no per-service work — mesh DNS (10.2.40.53) already
resolves <app>.renner.uno to that VIP; the only gap was that the
bgp-pool CIDR wasn't routable over the overlay.
Modeled on the cloudflared Deployment and held to the same in-cluster
firewall discipline: a dedicated CiliumNetworkPolicy
(netbird-ingress-egress) restricts egress to traefik + DNS + the NetBird
control/data plane only. Unlike the upstream NetBird operator the pod is
NOT privileged; the NET_ADMIN/SYS_ADMIN/SYS_RESOURCE caps are the
irreducible minimum for an L3 WireGuard routing peer.
- kubernetes/infrastructure/netbird-ingress/ (deployment, sops secret, kustomization)
- image-automation policy pinned to bare-numeric semver (excludes -rootless/-rc)
- netbird-ingress-egress CNP in network-policies/ingress-isolation.yaml
Requires one-time NetBird-side setup (dashboard/API): create a reusable
+ ephemeral setup key (auto-group k8s-ingress-router), replace the
placeholder in secret.sops.yaml, add a network route for 10.2.99.86/32
via that peer, and an access policy for the consuming mesh groups.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The rook-ceph HelmRelease's floating `>=1.16.0` constraint silently
upgraded to v1.20.0, which extracts CSI driver management into the
upstream ceph-csi-operator subchart. That subchart's generated
ctrlplugin Deployments and nodeplugin DaemonSets reference per-driver
ServiceAccounts (`rbd-ctrlplugin-sa`, `cephfs-ctrlplugin-sa`,
`rbd-nodeplugin-sa`, `cephfs-nodeplugin-sa`) that the main rook-ceph
chart no longer ships — they moved to a separate ceph-csi-drivers
chart. With the SAs missing, both ctrlplugin Deployments sat at 0/2
with FailedCreate for 5 days, blocking PVC attach/detach across the
cluster (Grafana, Alertmanager, libation stuck; Home Assistant pinned
to the edge node which never got an RBD nodeplugin pod).
The ceph-csi-drivers chart cannot be installed cleanly here:
helm-controller v1.5.4 predates `disableTakeOwnership`, so Helm would
refuse to adopt the operator-managed `Driver` and `OperatorConfig` CRs
the v1.18 -> v1.20 auto-migration created. Instead, render only the
SAs + RBAC the operator needs from the chart's templates as static
manifests under kubernetes/infrastructure/ceph-csi-drivers/. The
existing operator-managed CRs continue to be the source of truth.
Also pin the rook-ceph chart to `~1.20.0` to prevent the same
floating-constraint surprise on the next minor bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This reverts c7d6524 and 6b4610d.
The dummy0-based local-binding approach broke node InternalIP
detection. Cilium walks interfaces in index order and dummy0
(index 3) is lower than eth0/eth1, so once dummy0 had global-scope
LB IPs Cilium re-detected node IPs from there. This cascaded to
apiserver→kubelet routing breakage, pod IPs being allocated from
the wrong range, and eBPF state corruption that persisted past
agent restarts.
Binds every LoadBalancer Service's external IP as a /32 (IPv4) or
/128 (IPv6) on each node's dummy0 interface. Cilium's eBPF kpr
intercepts TCP/UDP for these IPs at the TC ingress hook before the
kernel's local-input decision, so local binding doesn't affect data
flow — it only makes the kernel respond to ICMP echo locally.
Why: Cilium L2 announcements and BGP both publish "I am this IP" to
the network without binding the IP locally on any host. ICMP echo
to LB IPs loops via the default route until TTL exhausts. kube-proxy
IPVS used to bind LB IPs on kube-ipvs0; we run full kpr instead.
Verified the binding+kpr composition works: bound 10.2.99.81 on
k8s-edge's dummy0 manually, ping returned 0% loss, TCP to 1883/8883
still hit the mosquitto backend, Shelly MQTT session unaffected.
Upstream Cilium PRs #37623/#40565 proposed an in-Cilium fix but
neither merged.
All ingresses migrated to traefik (last batch: loki). Remove the
HelmRelease + drop the ingress-nginx-egress CNP and the obsolete nginx
allowance from cloudflared-to-ingress-only. Other namespace CNPs that
list ingress-nginx as an allowed source are left in place — the
selectors don't match anything once nginx is gone; harmless cruft to
clean in a separate sweep.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ingress-nginx upstream archived 2026-03-24; migrate to Traefik for HTTP/3
support (LAN clients hit ERR_QUIC_PROTOCOL_ERROR against the nginx VIP).
Traefik runs alongside ingress-nginx (not default class) on 10.2.40.81 via
Cilium LB-IPAM sharing-key. shelly-manager is the pilot — manual UniFi DNS
flip 10.2.40.80 → 10.2.40.81 required to complete the cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spegel DaemonSet using each node's containerd content store, with Talos
machine.registries.mirrors routing all four mirrored registries at
127.0.0.1:30020. machine.files drops a containerd .part flipping
discard_unpacked_layers=false (Spegel prerequisite, containerd v3 plugin
path).
The LXC's Spegel rejection was specific to the ~10-rebuilds/day pipeline
iteration phase. With rebuilds now post-disaster only, a single rebuild's
docker.io pulls fit under the 100/6h anonymous ceiling, so the warm-cache
survivability argument no longer outweighs the operational cost of a
separate LXC.
Cutover is manual via per-node \`talosctl apply-config --config-patch
@kubernetes/talos/patches/registry-mirror.yaml -n <node> --mode=reboot\`
since discard_unpacked_layers needs a containerd restart. LXC 250
decommission follows in a separate PR once Spegel is verified serving
mirrored pulls.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alertmanager (kube-prometheus-stack) now sends to admin@renner.uno via
Proton SMTP submission (smtp.protonmail.ch:587 STARTTLS) using a per-
account token in the alertmanager-smtp SOPS-encrypted Secret.
A new infrastructure subtree, flux-notifications/, adds a Provider of
type alertmanager pointing at the in-cluster Alertmanager and two
Alert CRs: one info-severity for ImageUpdateAutomation bumps (with the
30m "no change" heartbeat excluded), and one error-severity catch-all
across every Flux source/sync kind. Flux events and Prometheus alerts
now share a single email funnel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a global cluster-settings ConfigMap (CLUSTER_IN_DEVELOPMENT,
ACME_SERVER) consumed via Flux postBuild substitution on the
infrastructure-crds Kustomization. A new letsencrypt-cluster
ClusterIssuer reads ACME_SERVER from the ConfigMap at reconcile time;
every Certificate / Ingress in the repo now references letsencrypt-
cluster instead of letsencrypt-{prod,staging} directly.
While CLUSTER_IN_DEVELOPMENT=true, ACME_SERVER points at acme-staging-
v02, so the only certs the cluster can issue are LE staging — sidesteps
the recurring 5-cert/168h prod rate-limit hit during back-to-back
rebuilds (latest occurrence: 2026-05-05 alloy-syslog + loki-ingress).
Flip both keys together when leaving development.
Per-app KSes intentionally do NOT enable postBuild — Grafana dashboards
and HA's deployment shell scripts contain unrelated \${VAR} markers
that would be clobbered. Substitution stays scoped to post-crds/ which
has no other variable expansions.
The standalone letsencrypt-prod / letsencrypt-staging ClusterIssuers
remain defined as escape hatches but are no longer referenced.
e2e-test.sh now also asserts letsencrypt-cluster Ready.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The prior agent's hostNetwork: true was the wrong abstraction. Earlier
in this PR I tried a Cilium-L2 LoadBalancer Service on 10.2.30.226,
but that only forwards a single TCP port (5580 ws) and does not give
the pod L2 presence on the IoT subnet — outbound is still cluster pod
network, no multicast, no Matter operational ports. Matter device
commissioning and discovery need real mDNS send/receive on the IoT L2.
Install multus-cni alongside Cilium and attach matter-server to a
NetworkAttachmentDefinition that does macvlan on k8s-edge's eth1 (the
v30iot bridge — verified via `talosctl -n k8s-edge get routes`,
10.2.30.0/24 is connected on eth1). Static IPAM pins 10.2.30.226 with
no gateway, so default-route traffic stays on the Cilium-managed
primary interface and only IoT-subnet traffic crosses net1.
The wiki note hostnetwork-app-workload-antipattern still applies, but
unlike the otbr precedent (commit 1972c9b) — where ClusterIP alone is
sufficient — matter-server actually does need IoT-VLAN reach, just not
via hostNetwork. multus is the documented escape hatch the wiki has
flagged but never exercised in this homelab; first install lands here.
iot-pool ledger now: .224 = home-assistant (LB), .225 = mosquitto
(LB), .226 = matter-server (multus, *not* a Cilium LB). LB Services
all pin explicit IPs via lbipam.cilium.io/ips so no auto-allocation
collides with the multus IP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds kubernetes/infrastructure/image-automation/ — central home for
Flux image-automation resources. The image-reflector + image-automation
controllers are already running (gotk-components.yaml ships them);
this is the first time we declare CRs that use them.
What lands:
- flux-system-write GitRepository (separate from the bootstrap
flux-system GR so we don't disturb its existing read-only auth)
pointing at gitea.renner.uno/gitea_admin/homelab via a Secret
named image-automation-git.
- ImageUpdateAutomation "homelab" walking ./kubernetes, applying
the Setters strategy from `# {"$imagepolicy": "..."}` markers,
pushing direct to master as security-bot@renner.uno.
- Pilot pair: ImageRepository + ImagePolicy "paperless-ngx" tracking
ghcr.io/paperless-ngx/paperless-ngx within "2.x" semver. The 3.x
cutover would need manual policy widening — by design.
- Marker on kubernetes/apps/paperless/paperless.yaml's image line
so the controller knows where to substitute.
- README explaining the bootstrap step and how to onboard more
images.
Manual one-time bootstrap REQUIRED before this works:
1. Create kubernetes/infrastructure/image-automation/git-secret.sops.yaml
from the template (token = security-bot Gitea PAT, write:repository
scope — same one used by the Woodpecker security_bot_token secret
can be reused).
2. sops -e -i to encrypt in place.
3. Uncomment the line in kustomization.yaml that lists it.
4. Push. Flux's existing infrastructure Kustomization handles the
sops-decryption.
Strategy choice (per user): direct push to master, not a PR branch.
Tight semver ranges per image are the safety net.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploys ori-edge/k8s_gateway (chart 2.4.0) watching Ingress resources
across the cluster and synthesising A/AAAA records pointing at the
ingress-nginx LB. Main CoreDNS delegates the renner.uno zone to it via
a pinned ClusterIP (10.96.0.11) — avoids DNS-name resolution during
CoreDNS startup.
Why: pods resolving *.renner.uno currently go via upstream DNS to the
legacy Caddy VIP (10.2.45.100), which silently routes server-side
traffic (OIDC token exchange, JWKS, webhook callbacks) to the old
prod stack instead of the in-cluster Service. Symptom: audiobookshelf
OIDC login fails with `invalid_grant` because the authorization code
issued by K8s Authentik is exchanged against legacy Authentik. The
wiki's `[[k8s-east-west-leak-after-host-split]]` names this failure
mode verbatim; the previously-attempted fix (`template IN A renner.uno`
synthesis) was too broad and got reverted in 7a09bad.
Shape of the fix:
- k8s_gateway watches only Ingress (resources: [Ingress]).
- `fallthrough renner.uno.` hands names without an in-cluster Ingress
to a `forward . 10.2.40.53` appended via extraZonePlugins — the
Proxmox DNS HA VIP serves everything else (gitea, immich, cloudflared
LXCs) unchanged.
- Main CoreDNS gets one `renner.uno:53` server block forwarding to
10.96.0.11. No wildcards, no manual FQDN list — adding a new app
Ingress just works.
External/browser DNS is untouched. No per-pod hostAliases needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cluster had no metrics-server, so the metrics.k8s.io API returned 404
and k9s columns stayed blank. Talos kubelets serve self-signed certs
(serverTLSBootstrap not enabled), so --kubelet-insecure-tls is required
and InternalIP is forced since nodes don't publish resolvable Hostname
addresses.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a Flux-managed ConfigMap that replaces the stock Talos CoreDNS
Corefile with one extra pair of template-plugin blocks: any A/AAAA
query for *.renner.uno is answered directly with the ingress-nginx
LoadBalancer IPs (10.2.40.80 / fd40::80). All other plugins (errors,
health, ready, log, prometheus, kubernetes, forward, cache, loop,
reload, loadbalance) are preserved verbatim.
Why this matters after the phase-2 app-config cleanup: K8s apps now
emit *.renner.uno hostnames. Without this synthesis, a pod doing a
server-side call to auth.renner.uno would hit cluster CoreDNS →
forward upstream → ingress.renner.uno CNAME → 10.2.45.100 (Caddy,
prod). That's an east-west leak into production. The template
forces every *.renner.uno lookup to stay inside the K8s LB. Nothing
left to leak.
Also adds a second Service (kube-dns-lb) in kube-system with the
same selector as the existing ClusterIP kube-dns, pinned via Cilium
LB-IPAM to 10.2.40.83 / fd40::83. Kasm workspace sessions can point
their Dns at this LB IP and get exactly the same view the cluster
pods have — no per-hostname ExtraHosts maintenance.
Existing kube-dns ClusterIP (10.96.0.10) is untouched so no pod
re-wires its /etc/resolv.conf.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The barman-cloud Deployment mounts barman-cloud-{client,server}-tls
Secrets that are created by cert-manager Certificates in post-crds. In
a single-Kustomization infrastructure reconcile this is circular: the
Deployment can't start without the Secrets, and post-crds can't run
until infrastructure reports ready. Move the plugin into its own
Kustomization that dependsOn infrastructure-crds so the Secrets exist
before the plugin Pod schedules.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vendored upstream manifest from
github.com/cloudnative-pg/plugin-barman-cloud releases/download/v0.12.0/manifest.yaml.
Installs the plugin controller into cnpg-system namespace with its
ObjectStore CRD (barmancloud.cnpg.io/v1) and cert-manager-issued
self-signed client/server TLS. cnpg-system namespace added to the
namespaces manifest.
Paired ObjectStore CRs + Cluster spec wiring for ak-pg / pprl-pg
come in a follow-up commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two blockers surfaced when the Velero HelmRelease reconciled:
1. velero namespace defaulted to `baseline` PSS — node-agent
DaemonSet needs hostPath mounts for /var/lib/kubelet/{pods,plugins}
and was rejected with PodSecurity violations. Labelled the
namespace `privileged` to match the rook-ceph/crowdsec/ingress
pattern.
2. The cluster had no `snapshot.storage.k8s.io` CRDs — Rook-Ceph
CSI does not ship them. Added the piraeus snapshot-controller
chart in kube-system; it provides the VolumeSnapshot,
VolumeSnapshotContent, and VolumeSnapshotClass CRDs plus the
controller that reconciles them. Required for Velero CSI
snapshot data movement and for our rook-ceph VolumeSnapshotClass
manifest to apply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Velero via HelmRelease in a new `velero` namespace, configured
for CSI snapshot data movement to the Garage S3 endpoint on nas-1
(k8s-velero bucket). Credentials SOPS-encrypted via the repo's age
recipient; AWS plugin loaded as an initContainer; node-agent enabled
for data movement and FSB fallback; native VSL snapshots disabled in
favour of the CSI path.
Paired with a Rook-Ceph VolumeSnapshotClass for RBD (default) and
CephFS — labelled for Velero auto-discovery. Adds a `velero`
namespace definition so Flux has somewhere to put everything.
No checksum gymnastics required on Garage today, but keeps the
aws-sdk-go-v2 `checksumAlgorithm=""` workaround documented in-place
against regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop duplicate HelmRepository external-dns in apps namespace (owned by
infrastructure/external-dns-unifi) to end SSA ownership flapping.
- Infrastructure Kustomization timeout 5m -> 15m; first-boot Rook +
cert-manager + cnpg + cilium do not fit in 5m on a cold cluster.
- authentik-db timeout 10m -> 30m; the restore job polls S3 for up to
10min and needs margin against the trigger-backup job.
- Remove orphaned sealed-secrets / external-secrets / metallb dirs that
were never referenced (Flux decrypts SOPS natively; Cilium handles L2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Automatically creates UniFi DNS records for all K8s Ingresses,
pointing to the ingress VIP. Replaces OpenTofu ingress_cnames.
- All Ingresses → UniFi DNS (internal, no annotation needed)
- Annotated Ingresses → Cloudflare DNS (external tunnel)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cloudflared is part of the ingress path, not an application.
Add SOPS decryption to infrastructure Kustomization for the
tunnel credentials secret.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move ExternalDNS from infrastructure to apps (not a prerequisite)
- Set --default-targets with tunnel ID so Ingresses only need
the annotation external-dns.alpha.kubernetes.io/cloudflare-proxied
- Simplify Gatus Ingress annotation (no tunnel ID needed)
- SOPS secret for Cloudflare token co-located with ExternalDNS
To expose via tunnel: add one annotation to Ingress.
To switch to NetBird-only: remove the annotation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ExternalDNS watches Ingress annotations and creates proxied CNAME
records in Cloudflare pointing to the tunnel endpoint.
- Deploy ExternalDNS with Cloudflare provider
- Annotate Gatus Ingress with tunnel target
- annotationFilter ensures only explicitly annotated Ingresses get
public DNS (opt-in, not opt-out)
- policy: upsert-only prevents deleting records not managed by K8s
To expose a service via Cloudflare tunnel, add to Ingress:
external-dns.alpha.kubernetes.io/target: <tunnel-id>.cfargotunnel.com
external-dns.alpha.kubernetes.io/cloudflare-proxied: "true"
To switch to NetBird-only: remove the annotations. No DNS record = VPN only.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace External Secrets Operator + Vault with SOPS-encrypted secrets
in git, decrypted by FluxCD using an age key.
- Add .sops.yaml with age public key for encryption
- Encrypt Cloudflare API token with SOPS
- Remove ESO HelmRelease and Vault ClusterSecretStore
- Add decryption config to infrastructure-crds Kustomization
- Bootstrap step: create sops-age secret from local age key
No external dependencies. Age private key lives on workstation,
injected during bootstrap via:
kubectl create secret generic sops-age -n flux-system \
--from-file=age.agekey=$HOME/.config/sops/age/keys.txt
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace SealedSecrets with Vault-backed External Secrets:
- New LXC 240 (vault.srv.renner.uno) for HashiCorp Vault
- Ansible role + playbook for Vault with Raft storage backend
- External Secrets Operator HelmRelease in K8s
- ClusterSecretStore pointing to Vault
- ExternalSecret for Cloudflare API token
- Remove SealedSecrets from infrastructure
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move CrowdSec to infrastructure stage with a Job that:
1. Waits for CrowdSec LAPI to be ready
2. Generates a bouncer API key via cscli
3. Creates a K8s Secret in the ingress namespace
ingress-nginx HelmRelease depends on CrowdSec, uses the CrowdSec
controller image, and reads the bouncer key from the provisioned
secret. No SealedSecrets or hardcoded keys needed.
Flow: CrowdSec deploys → Job creates bouncer key → ingress-nginx
starts with bouncer plugin authenticated to LAPI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cilium with kubeProxyReplacement=true has native LoadBalancer support
with L2 announcements. MetalLB L2 VIPs weren't reachable because
Cilium was handling the service datapath.
- Enable l2announcements and externalIPs in Cilium HelmRelease
- Add CiliumLoadBalancerIPPool (10.2.40.64-95) and L2AnnouncementPolicy
- Remove MetalLB from infrastructure
- Remove MetalLB annotation from ingress-nginx service
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lessons from manual validation run:
- Add privileged PodSecurity labels to metallb-system, rook-ceph, ingress namespaces
- Add Cilium HelmRelease with Talos-specific settings (cgroup, kubePrism)
- Consolidate all namespace definitions in namespaces/
- Update cert-manager to use crds.enabled instead of installCRDs
- Enable all infrastructure components in main kustomization
- Add Talos secrets to .gitignore (controlplane.yaml, talosconfig)
- Update node patches with correct install disk (vda) and zone labels
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
