First Service onto the new BGP-routed off-link LB pool. ha-vscode goes
from ClusterIP-behind-ingress to a direct LoadBalancer with an IP
auto-allocated out of 10.2.99.0/24 + fd99::/64 — exercising the
full path end-to-end: Cilium BGP advertisement → UDM-installed route
→ LAN client reachability.
Also tightens default-pool's serviceSelector to exclude `lb-pool=bgp`
labelled Services. Without this, default-pool's missing selector caught
every LoadBalancer and the bgp-pool allocation was undefined — the
smoke test had to fall back to a pinned `lbipam.cilium.io/ips`
annotation. With this fix, labelling alone is sufficient to direct
allocation, which is the migration pattern we want for the rest of the
Services.
ingress to vscode.home.renner.uno via traefik is unaffected (Service
keeps its ClusterIP; the LoadBalancer is additive).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The actual FRR config the UDM consumes — pairs with the cluster-side
cilium-bgp.yaml. Uploaded via UniFi UI → Settings → Routing → BGP as
bgp.conf. Lives in version control so the source-of-truth for what's
running on the UDM matches what's intended; the UI upload itself is
manual (no GitOps path to the UDM).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix design error in the BGP rollout. Control-plane traffic (BGP peering
between cluster nodes and the UDM) belongs on the management LAN, NOT
the IoT VLAN — regardless of where the kubelet InternalIP happens to
have landed. The IoT VLAN is the noisy device segment we're trying to
keep cluster infrastructure OFF of in the first place; putting BGP on
it would defeat the point.
peerAddress 10.2.30.1 → 10.2.40.1. Each node sources from its eth0
(10.2.40.x) via the kernel routing decision; no localAddress override
required. Matching FRR v3 lives in
~/Notes/raw/2026-05-19-udm-frr-bgp-config-for-cilium-v3.md and uses
`bgp listen range 10.2.40.0/24`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 of the BGP rollout — plumbing only, no service IPs change yet.
Cilium-cni: enable-bgp-control-plane=true. ipv4/ipv6 native-routing-cidr
stay UNSET intentionally; flipping native routing before the UDM has
BGP-learned routes would silently break pod→LAN egress. They land in
a phase 2 commit once peering is verified Established.
Cilium-bgp.yaml (new): BGPv2 CRs — CiliumBGPClusterConfig peering each
node with the UDM at 10.2.30.1 (cluster ASN 64513, UDM ASN 64512),
CiliumBGPPeerConfig with v4+v6 unicast SAFIs over a single TCP/IPv4
session, advertisements for PodCIDR and all LoadBalancer Service IPs.
The peering is over each node's v30iot InternalIP; the UDM side uses
`bgp listen range 10.2.30.0/24` for dynamic neighbors so node DHCP
churn doesn't break sessions.
Cilium-l2: bgp-pool added alongside default-pool / iot-pool, opt-in
via `lb-pool: bgp` label, range 10.2.99.0/24 + fd99::/64. Off-link from
every device VLAN — devices reach these IPs via the UDM's BGP-learned
routes, no on-link ARP and no GARP race exposure. Legacy pools + L2
announce policies stay active during phase 1 for soft migration.
Next steps (out-of-band, not in this commit):
1. Upload the matching FRR config to the UDM at Settings → Routing →
BGP (see ~/Notes/raw/2026-05-19-udm-frr-bgp-config-for-cilium-v2.md).
2. Verify BGP sessions Established from both sides.
3. Phase 2: add native-routing-cidr to cilium-cni.yaml.
4. Phase 3: migrate Services to bgp-pool, drop legacy pools + L2 announce.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Working theory: the multus/LB stack on edge's eth1 is implicated in
intermittent ESPHome / Voice PE TCP failures (Errno 113, 5s handshake
timeouts, RST). Test = remove HA's v30iot macvlan attachment and
measure aioesphomeapi WARNING/ERROR rate vs. the 7d baseline.
Trade-off accepted for the window: HA's built-in Shelly integration
loses zeroconf-driven host-field refresh, so Shelly entries may go
stale if a lease rotates during the test. matter-server, otbr, and
shelly-manager NADs are untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same Retain-policy leak as restore-test (commit 9a63232): every
invocation created throwaway PVCs against ceph-block and
ceph-filesystem, then deleted them. With reclaimPolicy: Retain
each delete left a Released PV behind. The cluster accumulated
6 such PVs over two recent runs before this fix.
Adds purge_released_pvs helper (jq-filtered claimRef match,
patch to Delete + remove) and calls it after both leak sites:
the ceph-block writer section (default/e2e-pvc) and the Velero
roundtrip section (velero-e2e/data, which is created twice per
run so both PVs need cleanup). Flipped the final two deletes to
--wait=true so PVs are reliably Released by the time we sweep.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous NAD assumed v30iot was eth2 on k8s-{1,2,3} based on a
mistakenly inverted comment. Empirical: `talosctl get links` on k8s-1
shows eth1 carries the virtio at bus_path=0000:06:13.0 (the v30iot
NIC) and eth2 is the X710 i40e (Ceph cluster network on VLAN 49). All
ARP / mDNS / HTTP probes from the pod were dropping at L2 because the
macvlan was sitting on the wrong physical interface.
Doesn't solve the PCI-probe-order fragility (Talos v1.12 has no
mechanism to rename the kernel `IFLA_IFNAME`; LinkAliasConfig only
sets a Talos-side label that upstream macvlan CNI doesn't resolve), so
that's left as a comment for a future CNI-side fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multus's `master:` is an interface name, and ethN ordering depends on
kernel probe order — on k8s-{1,2,3} the three NICs are mgmt virtio,
v30iot virtio, and the X710 i40e, and the assumption that v30iot
landed on eth2 wasn't holding: the macvlan was attaching to the X710
(VLAN 49 / Ceph cluster network), IPAM happily assigned 10.2.30.228
on it, and `connect: No route to host` followed for every Shelly
because ARP couldn't resolve anything on the wrong L2 segment.
Talos 1.12 ships `LinkAliasConfig` for exactly this — alias a physical
link by selector (bus path / MAC / driver) to a stable name usable as
the physical link name in every other networking config section. Add
the same alias on every node:
apiVersion: v1alpha1
kind: LinkAliasConfig
name: iot0
selector:
match: link.bus_path == "0000:06:13.0"
`0000:06:13.0` is the slot the tofu `extra_network_bridges = ["v30iot"]`
input always produces; verified consistent across k8s-edge and
k8s-{1,2,3} VM specs.
Then point every IoT-VLAN NAD at `master: iot0` so the same NAD works
on any node: home-assistant, shelly-manager, matter-server, otbr.
shelly-manager's stale `master: eth2` comment in its deployment.yaml
is updated to match.
Requires `talosctl apply-config` to each node — the v30iot link gets
bounced (down/rename/up) once on apply; mgmt + etcd traffic on the
default-route NIC are unaffected, but the v30iot L2 announce flap
will briefly drop the IoT-pool LBs (home-assistant-iot, mosquitto-iot)
for a few seconds per node, so sequence them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 7 Ingresses: remove leftover nginx.ingress.kubernetes.io/* annotations
(proxy-body-size / proxy-read-timeout / proxy-send-timeout /
proxy-buffering). Traefik streams without a body cap; the websecure
entrypoint idleTimeout is 3600s globally.
- 5 CNPs: drop inert `app.kubernetes.io/name: ingress-nginx` selectors,
keep traefik (or swap nginx→traefik for the paperless-ns east-west
egress rule). Comments updated.
- test-browser: remove the stale 10.2.40.80 hostAliases block (DNS
resolves naturally to the traefik VIP now).
- bootstrap.sh + e2e-test.sh: update local LB IP 10.2.40.80 → .86.
No functional change for live traffic — all removed bits were either
no-op on Traefik or referenced the retired nginx VIP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
All ingresses migrated to traefik (last batch: loki). Remove the
HelmRelease + drop the ingress-nginx-egress CNP and the obsolete nginx
allowance from cloudflared-to-ingress-only. Other namespace CNPs that
list ingress-nginx as an allowed source are left in place — the
selectors don't match anything once nginx is gone; harmless cruft to
clean in a separate sweep.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Traefik's BasicAuth Middleware rejects secrets with more than one
key: "found 2 elements for secret 'monitoring/loki-basic-auth', must
be single element exactly". Now that loki is fully on Traefik the
nginx-compatible `auth` key is unused; remove it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ingress-nginx auth-type/auth-secret/auth-realm trio becomes a Traefik
BasicAuth Middleware. Re-encrypts the SOPS Secret with both `auth`
(nginx convention, kept for safety during transition) and `users`
(Traefik convention) pointing at the same htpasswd content; SOPS file
now uses encrypted_regex so only stringData is encrypted (the rest of
the manifest is plaintext, matching the convention used elsewhere).
This is the last hostname on ingress-nginx. After this lands and
verifies, the controller can be decommissioned.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both Ingresses on the host flip together (path-based: `/` to
authentik-server, `/outpost.goauthentik.io` to the paperless-gate
outpost — Traefik picks the more specific match the same way nginx
did). authentik chart's nginx proxy-buffer-size/number drop because
Traefik streams.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three HA hostnames (home, home.k, vscode.home) flip to traefik. The
proxy sidecar in the HA pod stays the XFF trust anchor on the same
:8124/:8080 ports; the home-assistant-ns CNP already allowed traefik
on those ports (added during the shelly-manager batch).
Raises the websecure entrypoint read/write/idle timeouts to 3600s
globally to match the nginx proxy-read/send-timeout that HA dashboards,
vscode (code-server), woodpecker live log streams, and the pve noVNC
console rely on. Traefik's 180s default cuts long-lived idle WS.
cloudflared retargets home.renner.uno to the traefik Service; the .k.
and vscode.home. names are LAN-only via external-dns-unifi.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Traefik doesn't populate IngressRoute.status.loadBalancer, so
external-dns-unifi's traefik-proxy source had no implicit target and
silently produced no DNS records. Pin the target annotation to the
traefik Service VIP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The kubernetesIngress provider's service.serverstransport annotation
silently fails to resolve the apps/insecure-backend ServersTransport
CRD on Traefik 3.7 — file kept "working" only because its ExternalName
backend has a hostname-valid cert. IngressRoute references the
ServersTransport natively (services[].serversTransport: insecure-backend)
with no provider qualifier guessing.
external-dns-unifi gains the traefik-proxy source so it sees the
IngressRoute hosts (pve.renner.uno, browser.{,k.}renner.uno) and
publishes them into UDM DNS the same way it does Ingress objects.
Each app gets an explicit cert-manager Certificate (the
cert-manager.io/cluster-issuer annotation only works on Ingress).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Traefik's service.serverstransport annotation expects the
<name>@<provider> form per upstream docs. The bare name silently
resolved to nothing for ClusterIP-with-selector and EndpointSlice
backends (pve, test-browser) so the default insecure=false transport
was used and TLS failed on the self-signed backend cert. file
appeared to work only because its ExternalName backend
(nas.renner.uno) has a hostname-valid cert and was never relying on
insecureSkipVerify in the first place.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The kubernetesIngress provider resolves the
traefik.ingress.kubernetes.io/service.serverstransport annotation as a
bare ServersTransport name in the same namespace as the Ingress — the
<ns>-<name>@kubernetescrd qualified form is for IngressRoute CRDs.
With the wrong form, Traefik silently fell back to the default
transport and returned 500 on the backend TLS handshake.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each re-encrypts to a self-signed backend (NAS UI, PVE 8006, KasmWeb
Firefox 6901). Adds a shared apps/insecure-backend ServersTransport
with insecureSkipVerify, referenced from each Ingress via the
service.serversscheme / service.serverstransport annotation pair —
replacing the nginx backend-protocol HTTPS + proxy-ssl-verify off
pattern. test-browser drops its configuration-snippet (the
Authorization-header-strip workaround for the CrowdSec nginx fork's
HTTP/2 bug — Traefik does not have that bug).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apps with only proxy-body-size / proxy timeout annotations — Traefik
streams requests without an upper body limit by default and the timeouts
were nginx-specific to override the 60s default. cloudflared retargets
the three WAN-exposed names (audiobooks, immich, music). audiobookshelf
CNP now allows both ingress-nginx and traefik labels; music-assistant
and immich namespaces have no per-namespace CNP, so no policy change
there. Woodpecker's 3600s log-stream timeout maps to Traefik's
entrypoint idleTimeout — observing for WS drops before changing it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gatus (status), ical-proxy, kube-prometheus-stack (alerts/dash/prometheus)
have no nginx-only annotations, so the migration is pure
ingressClassName: nginx → traefik plus apps/allow-from-ingress now
permitting the traefik label as a source. cloudflared retargets
status.renner.uno (the only WAN-exposed name in this batch); the rest
are LAN-only via external-dns-unifi.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ceph-block and ceph-filesystem ship with reclaimPolicy: Retain so that
production PVC deletions can be undone manually. The restore-test has
the opposite need: every PVC it creates is throwaway and must not
survive the iteration. Without explicit cleanup each weekly run leaked
O(100 GiB) of CephFS subvolumes, filling the cluster to 95% (OSD_FULL,
HEALTH_ERR, restore writes rejected) on 2026-05-17.
cleanup_service now sweeps every Released PV whose claimRef belongs to
this iteration — destination ns PVCs, velero/ data-mover backup PVCs
(*-rtest-<RUN_ID>-*), and source-side snapshot probe PVCs (rt-src-*-<RUN_ID>) —
patching reclaimPolicy to Delete before removing so CSI purges the
backing storage. RBAC adds patch on persistentvolumes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
200 was being exhausted under HPA-scaled workload (server 2-4 + worker
2-4 + 3 outposts, each with connection pools).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
paperless ships <link rel="manifest"> in its HTML; browsers fetch the
manifest with credentials:'omit' per W3C spec, so the gate redirected
every page load to auth.renner.uno and burned an orphan session JWT.
The wider /static tree carries no auth value either — Django STATIC is
public by design; MEDIA stays gated. Pass through anonymously.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
paperless.renner.uno moves to traefik (cloudflared retargets); the
configuration-snippet host-redirect is replaced by a Traefik
RedirectRegex Middleware on a separate paperless-aliases Ingress
covering paperless.k.renner.uno and docs.k.renner.uno. authentik CNP
now allows ingress from traefik pods alongside ingress-nginx so the
gate outpost is reachable on its existing path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cilium socket-LB DNATs ClusterIP→pod at the socket layer, so the egress
policy sees the pod's container ports, not the Service ports. Traefik's
chart maps Service 443/TCP→containerPort 8443 (and 80→8000), so the
previous rule allowing 443/80 dropped every cloudflared→traefik SYN
with "Policy denied". Split the rule per backend.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cloudflared was locked to ingress-nginx only; docs.renner.uno requests
were timing out at the tunnel because cloudflared couldn't reach
traefik.ingress.svc. Add traefik to the same allowlist with the same
ports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Paperless's gate Service is an ExternalName pointing at the Authentik
outpost in the authentik namespace; Traefik blocks ExternalName backends
by default so the docs.renner.uno Ingress couldn't build its route. Match
the house pattern that ingress-nginx already uses.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First WAN-exposed cutover in the traefik migration. docs.renner.uno was
purely a 301 alias to paperless.renner.uno via configuration-snippet on
the shared 4-hostname Ingress; the redirect becomes a Traefik
RedirectRegex Middleware on a carved-off Ingress, and cloudflared swings
the hostname to traefik.ingress.svc. paperless.renner.uno and the .k.
aliases stay on ingress-nginx.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fauria/vsftpd has been unmaintained for years and is built on CentOS 7
(EOL 2024-06-30) — no base-OS security patches reach the image. The
replacement is gitea.renner.uno/k.renner/vsftpd, an ~8MiB alpine:3.21
+ vsftpd container under image-automation.
Aligning the vsftpd user with paperless's uid (1000:1000) also retires
the POSIX-ACL workaround in the init container: uploaded files come in
already owned paperless:paperless, so paperless's init-folders chown
is a no-op.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cilium LB-IPAM rejected the original pin because fd40::81 was already
held by rook-ceph/rgw-migration-lb. Bump to .86/fd40::86 — next free
slot in default-pool past the existing .80-.85 cluster.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ingress-nginx upstream archived 2026-03-24; migrate to Traefik for HTTP/3
support (LAN clients hit ERR_QUIC_PROTOCOL_ERROR against the nginx VIP).
Traefik runs alongside ingress-nginx (not default class) on 10.2.40.81 via
Cilium LB-IPAM sharing-key. shelly-manager is the pilot — manual UniFi DNS
flip 10.2.40.80 → 10.2.40.81 required to complete the cutover.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ori-edge/k8s_gateway is deprecated (last release Nov 2023, README
redirects to k8s-gateway/k8s_gateway). Chart name, fullnameOverride,
and service.clusterIP still supported, so this is a rolling upgrade.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The chart guards SM/PM template rendering on monitoring.coreos.com/v1
being present at install time. On a fresh-cluster bootstrap Velero can
install before kube-prometheus-stack lays down those CRDs, in which
case the chart silently skips the SM/PM and never re-renders until
values change — leaving the Grafana Velero dashboard permanently empty.
Move both objects out of the chart values and into Flux-managed
manifests. Flux retries on missing CRDs, so they always land regardless
of bootstrap ordering. Service still rendered by the chart so the
scrape target exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
paperless's s6 init-folders service chowns the entire consume tree to
paperless:paperless on every container start, so `chown -R 14:50` from
the ftp-scanner initContainer is reverted the next time paperless
restarts — leaving vsftpd's uid 14 with only other::r-x on
/scanner/Konstantin/ and producing 553 STOR errors (which the scanner
surfaces as "sending error").
Switch the strategy: own the tree as 1000:1000 (matching what paperless
maintains anyway) and grant vsftpd write access via a named POSIX ACL
entry u:14:rwx (plus default for new files). Named ACL entries are
preserved across chown, so paperless's recurring chown no longer breaks
FTP writes.
Also drop chmod -R 0755: chmod recomputes the ACL mask from the group
bits, which would cap u:14:rwx down to r-x and silently re-break writes.
kubectl --sort-by is silently dropped when paired with -o jsonpath, so
the previous form picked a non-deterministic Backup and could land on
one whose kopia snapshot had already been pruned by maintenance ("Unable
to load snapshot : snapshot not found"). Replace with a jq-side sort.
Immich library (~180 GiB, millions of small files) does not finish
within the prior 2 h per-restore budget; bump RESTORE_PHASE_TIMEOUT_S to
4 h and the cronjob activeDeadlineSeconds to 6 h so the full sweep can
complete after a long immich phase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Worker has been pinned at maxReplicas=4 for 45+ min despite steady-
state per-pod CPU of ~47m (47% of 100m request). Root cause: the
5-min scaleDown stabilization window picks the highest recommendation
across the window, so brief peaks above 70% during warmup phases
keep the HPA from ever scaling back down.
200m request puts the current steady-state at ~24% utilization,
well below the 70% target — HPA can scale back to minReplicas
between bursts, while still scaling up reactively when sustained
load exceeds 140m per pod.
Server unchanged: it's been steady at 24%/70% with 100m, no peaks
near target.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added a second `grafana.sidecar:` block at the
end of the grafana values, which collided with the existing one that
already configured `sidecar.dashboards`. YAML duplicate-key resolution
silently dropped one of them; the rendered Deployment showed no
resources on either sidecar.
Move the resources block inside the original sidecar map so both
keys coexist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both deployments ran the continuous reconcile loop with no requests,
which means they had no scheduling guarantees and would be first to
be evicted under node pressure. Live usage: 1m/150Mi each (memory
high vs CPU is the Go runtime's typical allocation profile).
Sized both at 25m/192Mi req, 384Mi limit. The unifi webhook sidecar
gets its own smaller block (10m/32Mi req, 64Mi limit — observed 1m/6Mi).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sized to current 2m/46Mi steady-state with headroom for ingest bursts:
20m CPU / 64Mi mem request, 256Mi mem limit. The config-reloader
sidecar already had requests from the chart defaults.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both had no resources block set. Sized to observed steady-state load
plus headroom:
- grafana main: 100m/768Mi req, 1Gi limit (was 28m/498Mi live).
- grafana sidecars (sc-dashboard, sc-datasources): 10m/128Mi req,
256Mi limit.
- kube-state-metrics: 50m/64Mi req, 256Mi limit (was 4m/28Mi live;
ksm scales with cluster object count, so generous headroom).
Note on ksm: kube-prometheus-stack's top-level `kubeStateMetrics`
section only toggles enable/disable. Resources must be passed via
the subchart name `kube-state-metrics:` (with hyphens).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Recreate caused every Flux image bump to fully stop ML before
starting the new version, leaving smart-search / face-detection
unavailable for the ~30s of model-loading cold start. The cache
PVC is RWX/ceph-filesystem so brief two-version coexistence during
rollout is fine.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.7
security-bot