Hermes Agent as a pod in its own `hermes` namespace: gateway (messaging
platforms, cron, OpenAI-compatible API) and dashboard, supervised by the
image's s6-overlay, state on a 10Gi ceph-block PVC.
- hermes.renner.uno (LAN): dashboard and Hermes Desktop remote gateway.
Plain Traefik plus the dashboard's own Authentik OIDC login (public PKCE
client, Hermes Users group) — a forward-auth gate would break Desktop's
Bearer-token calls.
- hermes-api.renner.uno (LAN): API server, Bearer key generated in-cluster
by an ESO Password generator.
- Read-only kubectl via the built-in `view` ClusterRole.
- CNPs: world 443/80, kube-apiserver, Traefik for LAN hostnames, and
Prometheus/Alertmanager/Loki (with the matching monitoring-side grant).
- Image automation for nousresearch/hermes-agent v-tags.
After deploy: log in to Anthropic inside the pod
(`hermes auth add anthropic --type oauth --no-browser`, then `hermes model`);
platform and Home Assistant tokens go in via the dashboard.
Not yet covered: a Velero schedule for the PVC.
Namespace, CRD (Flux prune disabled so a prune never cascades into
deleting every Kuma monitor via finalizers), cluster RBAC matching the
controller-gen role, Recreate Deployment of the first master build
(2026-10-19-0545) under image automation, and a default-deny
CiliumNetworkPolicy allowing only DNS, the kube-apiserver and
kuma.renner.uno:443 out, kubelet probes in.
The Kuma login comes from its own OpenBao path infrastructure/kuma-sync
(a copy of the single Kuma admin account) rather than reading
infrastructure/uptime-kuma.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AutoKuma 2.0.0 never applied its ownership tag against Uptime Kuma 2.5.6, so a pod restart with empty /data duplicated every managed monitor (pilot step 5). Replaced by an in-house controller under a separate plan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AutoKuma 2.0.0 (Kubernetes source) syncs KumaEntity objects to Uptime Kuma
at https://kuma.renner.uno. Ships no KumaEntity objects. Image pinned, not
under Flux automation: Kuma's internal API is unofficial, so Kuma and
AutoKuma are bumped together.
ClusterRole is cluster-wide because the controller lists/watches
KumaEntity across all namespaces. The openbao ClusterSecretStore
namespace allowlist gains autokuma. FILES and DOCKER sources are
disabled; /data and /tmp are emptyDirs for readOnlyRootFilesystem.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes the Paperclip deployment (namespace paperclip, CNPG cluster
pclp-pg, ingresses, image automation, server image build) and the
infrastructure that existed only for its agents:
- agent sandbox: paperclip-renner namespace and CNPs, the
kubernetes-sigs/agent-sandbox controller with its namespace, CNP and
Renovate rules
- agent-runtime-{go,ops} images and the build-agent-runtime pipeline
- matrix bridge (Element <-> Paperclip agents) and synapse's appservice
registration mount
- monitoring loki-read-gateway (only called by the paperclip Role)
- MQTT account agents-read and the controller's RBAC in ns paperclip
- the OpenBao secret-exists-check policy file
- agent admits in the wiki, lidarr, shelly-manager, mosquitto and
hubble-relay CNPs; the paperclip-api Gitea webhook host; the wiki API
key; the ClusterSecretStore namespace entry; the pclp-pg RECOVERY row
- the Paperclip design spec and plan
Authentik group "Paperclip Users" is switched to state: absent so the
blueprint deletes it. package-retention's EXCLUDE_PACKAGES becomes
optional, since its only value was the agent-runtime prefix.
The shelly-manager api-proxy sidecar stays: mqtt-account-controller
calls shelly-manager-api too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Namespace, Deployment (one replica, scratch image, health and readiness probes), Service, ExternalSecret (webhook secret from apps/review-bot, Mistral key from apps/wiki, the rotated review-bot token), CiliumNetworkPolicies (Gitea in, Traefik and api.mistral.ai out), the namespace in the ClusterSecretStore allowlist, Gitea's egress to the service and its webhook host list, and Flux image automation for the review-bot image.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
OpenProject Community hides every OIDC provider without an Enterprise
token (auth_plugin.rb, EnterpriseToken.allows_to?(:sso_auth_providers)),
so it cannot sign in through Authentik.
Removes the app, its Flux Kustomization, namespace, network policy,
op-pg PodMonitor, blackbox probe, Velero schedule and ClusterSecretStore
entry. Authentik drops the OIDC secret env and marks the OpenProject
application, provider and group absent. Gitea no longer allows
openproject.renner.uno as a webhook host.
Refs: REN-1363
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New openproject namespace and Flux KS: web + worker pod (17.9.1-slim,
digest-pinned), op-pg CNPG 17 with Barman to Hetzner, attachments PVC
with a Velero schedule, ESO-generated SECRET_KEY_BASE and admin password,
public Ingress on openproject.renner.uno via the NetBird edge, default-deny
CNP, op-pg PodMonitor and a blackbox health probe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New `matrix` namespace and Flux KS: Synapse v1.162.0 on a CNPG Postgres
(mx-pg, Barman to Hetzner), media store + signing key on synapse-data
(Velero matrix-daily), Element Web locked to matrix.renner.uno, and an
idempotent Job that creates the one owner account from OpenBao
apps/matrix. Federation off, registration and guests closed, LAN-only
ingress, egress limited to the in-cluster DB and the push gateway.
Appservice slot left empty for REN-565 Part 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- infrastructure/agent-sandbox: Flux GitRepository (tag v1.0.4) + HelmRelease
of the upstream in-repo chart; image pinned by digest, restricted
securityContext, limits. CRDs via crds: CreateReplace.
- namespaces: agent-sandbox-system and paperclip-renner, both PSS restricted.
- network-policies: CNP for agent-sandbox-system (probes in, apiserver out).
- apps/paperclip/sandbox-rbac.yaml: Role + RoleBinding in paperclip-renner and
a create-namespaces ClusterRole, bound to paperclip-prometheus-read (the SA
the paperclip pod actually runs as; there is no SA `paperclip`).
- renovate.json: group the git tag and the image pin into one PR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- apps/renovate: CronJob (daily 04:00, routine PRs Monday only via the
shared preset's schedule), runner config, renovate-bot token ExternalSecret.
- clusters/homelab/renovate.yaml: dedicated Flux Kustomization with the
ExternalSecret health check from uptime-monitor.yaml.
- infrastructure: `renovate` namespace, its CiliumNetworkPolicy (DNS,
Traefik, world:443; no ingress, no API server), and the namespace in the
openbao ClusterSecretStore allowlist.
- renovate.json: extends local>k.renner/renovate-config. Managers flux,
kubernetes, woodpecker, dockerfile. Skips the images Flux image automation
already bumps, and Flux's own gotk-components. Digest pins and
Cilium/Rook-Ceph minors get their own PRs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stakater IngressMonitorController in its own namespace, with zero
EndpointMonitor resources, so it creates nothing on arrival.
- Vendored EndpointMonitor CRD from chart 2.2.13, image pinned to v2.2.13
and deliberately kept out of Flux image automation.
- Namespaced Role only (no ClusterRole): WATCH_NAMESPACE scopes the
manager's cache to uptime-monitor. Verbs derived from what v2.2.13
actually calls; every omission is justified inline in rbac.yaml.
- The ExternalSecret now renders IMC's config.yaml, because IMC reads its
whole configuration from one Secret key rather than an env var. Config
stays in git; only the key is injected.
- Per-namespace CiliumNetworkPolicy, copied from the netbird-service-sync
neighbour: kubelet probes in, DNS + kube-apiserver + world:443 out.
- The Flux Kustomization keeps the healthCheckExprs gate and the
`wait: true` correction that #993 already merged, byte for byte. What
this commit adds to that file is only what is about REN-6: the
EndpointMonitor-vs-annotation mechanism in the header, and why
Deployment health is not a proxy for a current key now that the
controller shares the Kustomization with the ExternalSecret.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Makes the key at OpenBao `infrastructure/uptime-robot`, field `api_key`,
available as a Secret in the new `uptime-monitor` namespace and nowhere
else. No new operator: this uses the External Secrets Operator and the
`openbao` ClusterSecretStore the cluster already runs.
Nothing is monitored by this commit. The namespace holds only the key;
the IngressMonitorController, its ServiceAccount and its RBAC arrive
separately (REN-6).
* `data` + `property` instead of `dataFrom: extract`, so only the one
field lands in the cluster rather than every field at the KV path.
* Adds `uptime-monitor` to the ClusterSecretStore `conditions.namespaces`
allowlist, sorted. Without this ESO reports "not allowed to reference
ClusterSecretStore" and the Secret silently never syncs.
* A DEDICATED Flux Kustomization rather than adding the ExternalSecret to
infrastructure/post-crds. That path runs `wait: true` with no explicit
healthChecks and 19 other Kustomizations dependOn it, so an
ExternalSecret whose OpenBao path is not yet populated would never turn
Ready and would stall most of the cluster. Here the blast radius of a
missing key is this Kustomization alone, which nothing depends on.
* No OpenBao policy change: eso-read already grants `homelab/data/*`.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Navidrome (Subsonic API) as the Spotify replacement for phone apps.
- Library PVC on ceph-filesystem-bulk (RWX, 300Gi, mounted read-only);
SQLite data on ceph-block (RWO).
- Web UI behind a dedicated navidrome-gate Authentik proxy outpost;
Navidrome trusts X-Authentik-Username (ExtAuth) and auto-creates the
user on first SSO login (first user = admin).
- /rest and /share routed straight to Navidrome with the identity header
stripped, so Subsonic clients use their Navidrome password.
- Namespace baseline CNP, gate egress CNP, Flux image automation,
Velero daily schedule (09:15), bootstrap wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Additive; the running controller is untouched:
- ci/images/netbird-proxy-services -> ci/images/netbird-service-sync,
build step and image name renamed, so the merge builds
gitea_admin/netbird-service-sync.
- Namespace netbird-service-sync with a baseline policy: probes in;
out only to kube-dns, kube-apiserver, OpenBao :8200 and world :443
(NetBird management API).
- openbao-ns admits netbird-service-sync on :8200, like wiki-index.
- policies/netbird-service-sync.hcl documents the new policy and role
(applied by hand, as before).
Refs #876
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Additive only, nothing moves yet:
- Namespace gitea, declared in infrastructure like woodpecker.
- network-policies/gitea-ns.yaml: baseline (same-ns, monitoring, host,
ICMP; egress to traefik, kube-dns, wiki-index, world) and
gitea-allow-traefik for :3000 and :2222.
- openbao rotator egress (flux-git, review-bot) and wiki-index ingress
also admit gitea in the gitea namespace; the apps entries stay until
stage 3.
- Velero schedule gitea-daily (07:45) so data-gitea-0 stays backed up
after the move; apps-daily comment updated (Woodpecker left apps).
Refs #878
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found by rebuild rehearsals; each stopped a cold start outright and none
changes behaviour on a running cluster.
infrastructure-ceph-cluster no longer dependsOn infrastructure. That
edge closed a cycle no new cluster could escape: infrastructure is Ready
only once its openbao healthCheck passes, openbao-0 runs only once its
PVC binds, and the PVC wants the ceph-block StorageClass defined in this
very Kustomization. infrastructure runs wait: false, so everything this
path needs is applied long before readiness; on an empty API server it
self-heals on retryInterval like post-crds already does.
home-assistant gains dependsOn infrastructure-cnpg-barman: its
ObjectStore kind arrives with the Barman plugin, and one unknown kind
fails the server-side dry-run for the whole apply set — taking
speech-to-phrase, zone-configurator and matter-server down with it.
Every other ObjectStore user already declares the edge.
woodpecker-ci namespace moves to infrastructure/namespaces: its
CiliumNetworkPolicies ship with infrastructure-crds, which runs before
every app KS, so declaring the namespace in the workload KS inverted the
order and held infrastructure-crds — and the twenty app Kustomizations
behind it — on "namespaces woodpecker-ci not found". Same reasoning as
its sibling namespace, previously not carried across.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The evaluation is over. Removing clusters/homelab/paperless-eval.yaml is
the designed teardown: that Kustomization is prune: true with no
prune-disabled annotation anywhere underneath, and its own comment says
being able to delete the namespace by removing the file is the point.
Removed alongside it, so nothing recreates the namespace or points at it:
kubernetes/apps/paperless-eval/ the manifests
clusters/homelab/paperless-eval.yaml the Flux Kustomization
network-policies/paperless-eval-ns.yaml and its kustomization entry
namespaces/namespaces.yaml the Namespace object
network-policies/ai-ns.yaml the llama-embed ingress grant
post-crds/openbao-clustersecretstore.yaml the ESO namespace grant
Production backups are unaffected. The eval Cluster read
s3://renner-uno-k8s-cnpg/pprl-pg as a recovery source with
isWALArchiver: false and no retentionPolicy, so it never wrote to the
production prefix and deleting it removes nothing from the store.
The model weights on the namespace's PVCs were already deleted by hand
before this change; the volumes go with the namespace.
scripts/paperless-eval/ is deliberately left in place — it is the tooling
that drove the evaluation, not part of the namespace, and it is only
useful if the namespace is ever recreated from these manifests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Choosing a chat model, and later an embedding model, means running the
same 50 documents through each candidate. That has been happening against
the live archive: the embedding model was swapped under it, the LLM index
rebuilt twice, the chat server broken for an afternoon, and the UI left
spinning while an eval held the only GPU slot. None of that belongs there.
The database is 90 MB and the AI path never opens a file — it reads
title, content and filename out of Postgres — so a copy costs a restore
and an empty media volume rather than a second copy of the documents.
Restores from paperless's backup store and archives nothing back. Three
omissions carry that, rather than any setting pointing somewhere safe:
no spec.plugins the cluster has no barman plugin at all, so no
WAL is archived
no retentionPolicy the source ObjectStore points at the production
prefix; with a retention policy this namespace
could expire paperless's backups
no ScheduledBackup nothing here is worth keeping
pg_basebackup would have avoided touching the store at all, but it needs
the production cluster's credentials, and those live in a Secret in the
paperless namespace that this one cannot read.
Everything that exists to serve people is left out: no ingress, no gate
outpost, no hostname, no remote-user auth, no Tika or Gotenberg, no mail
or FTP consumption. Nothing signs in and nothing is consumed, so the
instance is unreachable from outside the cluster and the documents are
not exposed a second time.
The deployment here is a smoke test — it proves the restore is usable and
that the search index builds. The evaluation runs as jobs that bring
their own paperless and llama-server, since a paperless instance can only
point at one chat endpoint and two candidates need to run at once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Backing up an external server is that server's own business. The pull path
made the cluster responsible for it and, worse, put a root key on the VPS
that the cluster held — a standing inbound grant that existed only to move
a SQLite file.
Removed cluster-side: the netbird-backup namespace and everything in it
(the nightly CronJob, the netbird-backup-ssh ExternalSecret, the 5Gi
ceph-block PVC), the namespace's entry in the OpenBao ClusterSecretStore
allow-list, the netbird-backup-daily Velero Schedule, and the
NetbirdVpsBackupStale / NetbirdVpsBackupNeverRan alerts.
Flux prunes the PVC, which destroys the 14 retained dumps (2026-08-15
through 08-28). That is intended. Velero's existing offsite copies age out
on their own under the schedule's 720h TTL; no cleanup is needed there.
Removed VPS-side: the dump script, and the forced-command key authorising
it. Deleting the two tasks that placed them would only stop ansible
managing them and leave both live on the host, so they are replaced with
state: absent tasks that actually revoke them. Those two can go once the
role has run against every host that ever carried them.
The trmnl-daily schedule's stagger comment pointed at netbird-backup-daily
for its slot; it now points at apps-daily, which is the schedule that
actually precedes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds llama.cpp's server in router mode as the cluster's OpenAI-compatible
inference endpoint, in a new `ai` namespace, running entirely on the on-board
Iris Xe via the `gpu.intel.com/i915` resource.
One Deployment serves both a chat model and an embedding model: started with
--models-preset and no -m, llama-server supervises each INI section as its own
subprocess behind a single port, routed by the request body's "model" field.
That gives /v1/chat/completions, /v1/embeddings and /v1/rerank from one pod,
with a crash in one model not taking out the other.
Backend and build are chosen from measurement rather than documentation. On
this hardware (i9-12900H, Iris Xe 96 EU) Vulkan beats the SYCL/oneAPI build by
+50% prefill and +36% decode, so the Vulkan image is pinned to b10731 — the
exact build benchmarked. Full offload (-ngl 99) matches CPU throughput at
~10.8 tok/s decode and ~137 t/s prefill while leaving the node's 14 cores free
for the rest of the cluster; the iGPU is here for that isolation, not speed.
The weights are fetched by an initContainer on first start, so no manual
seeding step stands between a merge and a working endpoint. They land on a
prune-disabled CephFS RWX PVC and are skipped on later starts.
Exposed LAN-only at llm.renner.uno: the API is unauthenticated, so the
`dns.renner.uno/public` label is deliberately omitted and only the UDM
external-dns instance publishes the record.
The pod needs no hostPath and no privilege — the device plugin passes the
render node in through the device-plugin API — so `ai` takes the cluster's
default `baseline` enforcement with no pod-security labels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses the third blocking review finding on #504.
The PR claimed the namespace move fixes wp-pg's absent metrics. It does not,
on its own. Prometheus requires the `release: kube-prometheus-stack` label
that CNPG's own enablePodMonitor omits, so every scraped cluster has a
hand-written PodMonitor in apps/monitoring-rules/cnpg-podmonitors.yaml —
ak-pg, im-pg, gf-pg, pprl-pg. There has never been one for wp-pg.
So wp-pg was missing both halves: no scrape target configured, and no
monitoring ingress allow in the apps baseline. Fixing the policy half was
necessary but not sufficient, and the runbook's cnpg_collector_up check
would have returned empty either way — unable to confirm the thing it
existed to confirm. Adds the PodMonitor in the woodpecker namespace.
Also from the review:
- Added a ScheduledBackup for the restored cluster. Every other CNPG
cluster here pairs isWALArchiver with one. Without it the new
wp-pg-woodpecker prefix takes continuous WAL but never a base backup, so
it has no independent recovery point and wp-pg-store's 30d retention has
nothing to prune against — the WAL chain grows unbounded for as long as
stage 1 sits before cutover. Scheduled 02:40, clear of the apps cluster's
02:35 and the other clusters' 02:30.
- Documented that the `woodpecker` KS goes not-Ready while the restore runs.
Its healthCheckExprs matches Cluster by GVK with no name qualifier, so
Flux evaluates it against both the live apps/wp-pg and the restoring
woodpecker/wp-pg, and wait: true makes the KS that owns production
Woodpecker report unhealthy. Expected, does not cascade since nothing
dependsOn it, but it would read as an outage without a note.
- Dropped the app.kubernetes.io/name label from the new Namespace; no other
namespace in that file carries one.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the blocking review finding on #504.
The "step pods call the server back on :8000" rule was attached to the
namespace baseline, which uses endpointSelector: {}. A toPorts block
narrows the port, not the destination workload — that is governed solely by
the enclosing policy's endpointSelector. So the rule granted woodpecker-ci
access to :8000 on every pod in the namespace, and the CNPG instance
manager serves its status API on :8000 (server.yaml:94 uses the same port).
Arbitrary repo-supplied build steps would have had a route to wp-pg's
instance manager.
That is the same endpointSelector:{}-means-blanket-grant failure this PR
exists to escape, reproduced inside the fix.
Removed rather than rescoped: there is no woodpecker server in this
namespace until stage 2, so the rule was premature as well as wrong. It
lands there as its own policy selecting the server pod, the way
woodpecker-allow-rotators already scopes its :8000 rule.
The baseline now carries no port-scoped rules at all, so nothing in it can
be mistaken for workload scoping, and a comment warns why that must stay
true.
Also from the review:
- Corrected the KS attribution. The network policies ship with
infrastructure-crds (infrastructure/post-crds includes
../network-policies/), not the top-level infrastructure KS. The ordering
conclusion is unchanged — infrastructure-crds dependsOn infrastructure,
and woodpecker dependsOn both — but the comments named the wrong one.
- Documented why the baseline has no egress section, instead of leaving it
to read as an oversight. Writing that ruleset in stage 2, with the server
and agent present, beats guessing it now and breaking the WAL archiving
this stage exists to verify.
- Dropped the monitoring -> :9187 rule from wp-pg-allow-cnpg-operator. The
baseline already allows the monitoring namespace unrestricted, so it added
no coverage while implying scoping it did not provide.
- Added a cilium-dbg policy trace step to the runbook. dry-run validates
manifest shape, not Cilium semantics, and would not have caught this bug.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #433. No cutover, no CI impact: this stands up the `woodpecker`
namespace and restores wp-pg into it from barman, running in parallel with
the live apps/wp-pg. Nothing points at it yet.
Why a namespace move rather than a policy fix. wp-pg has been Ready=False
with "Instance Status Extraction Error" for three weeks because the CNPG
operator in `database` cannot reach https://<pod-ip>:8000/pg/status. The
apps default-deny-ingress uses endpointSelector:{}, so the only
namespace-level remedy is to allow the whole database namespace in — which
would expose gitea, gatus, external-dns and ical-proxy along with wp-pg, to
fix one pod on one port. The same gap explains why cnpg_collector_up has
series for authentik, immich, monitoring and paperless but not wp-pg: apps
never allowed the monitoring namespace either.
woodpecker-ci is deliberately not the target. It enforces pod-security:
privileged and exists for ephemeral step pods running repo-supplied code;
the database has no business sharing it.
Two ObjectStores, deliberately. wp-pg-store-source reads the existing
s3://renner-uno-k8s-cnpg/wp-pg prefix; wp-pg-store archives to a new
wp-pg-woodpecker prefix. Two clusters writing one destinationPath would
interleave their WAL streams and corrupt both, and splitting them keeps the
pre-migration backup intact as rollback.
The Namespace is declared in infrastructure/namespaces rather than beside
these workloads: its network policies ship with the `infrastructure` KS,
which the `woodpecker` KS dependsOn, so creating it in the workload KS would
invert that order and deadlock a cold rebuild.
The restored cluster deliberately carries no prune: disabled guard. The
original has one because it holds the only copy of the data; this one is
reconstructable, and a prune guard on a migration artifact is how orphans
outlive an abandoned migration. It gains the guard in stage 3.
Barman creds still resolve from seed/paperless/hetzner-creds. #397 wants a
woodpecker-owned path and this was the natural moment, but creating it needs
OpenBao write access that is not available here (bao kv list seed -> 403).
Pointing at a path that may not exist would leave the restored cluster
unable to archive WAL, which is what this stage exists to prove.
Runbook, including how to verify the restore holds real data rather than
merely coming up healthy: MIGRATION-namespace.md.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
WordPress speaks MySQL natively. The PostgreSQL route would mean pg4wp, an
SQL-rewriting shim whose README lists WordPress 6.5.3 as its newest tested
version while renner.uno runs 7.0.4 — every core update would become a
compatibility gamble.
Charts are OCI artifacts, so this follows the OCIRepository pattern from
spegel rather than the HelmRepository pattern used for CNPG. CRDs ship as a
separate chart that upstream requires first, hence the dependsOn. The version
range is capped at the year boundary: the project uses calendar versioning and
ships an upgrade guide with each minor.
renner-uno-k8s-mariadb carries the same 30d noncurrent-version expiry as the
CNPG and Velero buckets, matched to the Backup resource's maxRetention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Phase 1 of the TRMNL X dashboard plan: stand up the self-hosted Terminus
(BYOS) server the panel will poll. The Home Assistant dashboard, the e-ink
theme and the screenshot renderer are later phases and are not in here.
Phase 0 verification, all confirmed against upstream before writing any of
this:
* TRMNL X IS a first-class model in Terminus (id v2 / label "TRMNL X",
1872x1404, 16 colours, 4-bit) — the stop-and-report condition in the
plan does not trigger. Note it is NOT in config/db/seeds.rb: the model
list is populated by the MODEL_SYNCHRONIZER job pulling trmnl.com, so
the Models list is empty until that job has run once.
* The HA add-on can run standalone. Our HA is plain HA Core, not
Supervised, but the add-on reads env vars in preference to
/data/options.json and authenticates with an ordinary long-lived
token — SUPERVISOR_TOKEN only flips an isAddOn boolean. So the
fallback path in the plan (upstream container, not add-on) is viable.
That deployment lands with Phase 4.
Deviations from the repo's house patterns, each deliberate:
* tr-pg runs PostgreSQL 18, not the standard 16. Terminus ships a
committed config/db/structure.sql containing `SET transaction_timeout
= 0;`, a GUC that does not exist before PG 17, so schema load against
a 16 server fails outright. This is the first non-16 CNPG cluster
here. The existing 16 pin is inherited from 739a67e9 (Apr 2026) and
has no documented rationale.
* The device endpoint is cleartext HTTP on a Cilium BGP LoadBalancer
(trmnl-device.renner.uno) rather than an Ingress. The firmware cannot
be given the internal CA, and a TLS-less Ingress would be a new
pattern for this repo — a dedicated LB address is not. The admin UI
keeps a normal TLS Ingress at trmnl.renner.uno; both front the same
Service. Same split as home.renner.uno / ha.renner.uno.
* ghcr.io/usetrmnl/terminus is NOT enrolled in image automation despite
publishing a usable tag stream. It serves the firmware and
/api/display contract to a wall-mounted device, upstream is pre-1.0
with no releases or changelog, and an unattended bump that breaks
either leaves the panel dark. Bumps are manual and reviewed. The
Valkey sidecar is enrolled — it touches no device-facing contract.
* Egress is an FQDN allowlist (trmnl.com, plus the S3 host that serves
firmware binaries), not toEntities: world. The S3 host is returned by
the API at request time rather than being a source constant, so it can
drift; the policy comment says where to look when firmware sync
starts failing. This also means Terminus's Extensions/Poll feature,
which fetches admin-configured URLs, will not work — accepted, since
screen content is pushed in by the renderer.
* The CNPG pods are deliberately left OUT of that egress policy so the
barman WAL archive path is not behind a rule that fails closed.
Images verified to exist in their registries before pinning:
ghcr.io/usetrmnl/terminus:0.68.0, valkey/valkey:9.1-alpine,
ghcr.io/cloudnative-pg/postgresql:18.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes the last big SPOF-assessment item: the management SQLite store
(peers, setup keys, groups, routes) had zero backup coverage — losing
the Hetzner VPS meant re-enrolling the entire mesh. The VPS cannot
reach any PBS (both PBS names resolve to the public proxy, ports
closed), so the cluster pulls instead of the VPS pushing:
- VPS (netbird-server role): a dump script takes a CONSISTENT snapshot
via sqlite3 .backup (online-backup API, safe against the live writer)
plus /opt/netbird/config and streams a tar.gz to stdout. A dedicated
ed25519 key is authorized for root restricted to exactly that script
(restrict,command=…) — the puller can run nothing else.
- Cluster (kubernetes/infrastructure/netbird-backup): nightly CronJob
(02:45) ssh-pulls the dump onto a 5Gi Ceph PVC, gzip-verifies before
renaming, keeps 14 dumps; ExternalSecret syncs the private key from
OpenBao (homelab/infrastructure/netbird-backup, one-time manual
provisioning like the rotator bootstraps); restricted-profile pod.
- Offsite: new Velero Schedule netbird-backup-daily (07:45, PVC-only,
snapshotMoveData, 30 d TTL) ships the PVC to Hetzner Object Storage.
- Alerts: NetbirdVpsBackupStale (critical, >26 h without success) and
NetbirdVpsBackupNeverRan (warning — catches the unprovisioned-secret
state so the backup cannot die silently).
Restore: untar over /opt/netbird/{management,config} after the role
rebuilds the stack.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Single-node raft OpenBao (5Gi ceph-block, UI at openbao.k.renner.uno,
internal-only ingress) and ESO 2.x. No ClusterSecretStore yet — it can
only turn Ready after manual init/unseal, and committing it now would
freeze the infrastructure-crds wait gate (2026-07-13 incident shape).
Full phased plan incl. per-file checklist and DR runbook in
kubernetes/MIGRATION-sops-openbao.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
Deploys evcc (evcc.io) as a monitoring-only instance — no charger or
loadpoint yet — to validate the Vitocharge VX3 -> open3e -> MQTT energy
pipeline and get familiar with evcc ahead of a future wallbox.
evcc runs as a plain Deployment (house style, like gatus) with:
- grid/PV/battery custom MQTT meters + battery SoC, reading the VX3
open3e datapoints (grid 1603, PV 1690, battery power 1836, SoC 1664)
- config delivered via a sops-encrypted Secret (mqtt password inline)
- Traefik ingress on evcc.renner.uno / evcc.k.renner.uno
- a 1Gi ceph-block PVC for evcc's SQLite state
- a dedicated Flux Kustomization (clusters/homelab/evcc.yaml)
- a per-namespace CiliumNetworkPolicy (evcc-ns.yaml)
MQTT: adds a scoped broker user `evcc` (read-only on open3e/#) to the
Mosquitto deployment, plus the reciprocal home-assistant-ns ingress rule
so evcc can reach mosquitto:1883 via ClusterIP. The password is a shared
sops value (evcc namespace + home-assistant namespace).
Note: applying this restarts the Mosquitto pod once (brief MQTT blip for
HA/open3e), because the broker's env/command changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spegel DaemonSet using each node's containerd content store, with Talos
machine.registries.mirrors routing all four mirrored registries at
127.0.0.1:30020. machine.files drops a containerd .part flipping
discard_unpacked_layers=false (Spegel prerequisite, containerd v3 plugin
path).
The LXC's Spegel rejection was specific to the ~10-rebuilds/day pipeline
iteration phase. With rebuilds now post-disaster only, a single rebuild's
docker.io pulls fit under the 100/6h anonymous ceiling, so the warm-cache
survivability argument no longer outweighs the operational cost of a
separate LXC.
Cutover is manual via per-node \`talosctl apply-config --config-patch
@kubernetes/talos/patches/registry-mirror.yaml -n <node> --mode=reboot\`
since discard_unpacked_layers needs a containerd restart. LXC 250
decommission follows in a separate PR once Spegel is verified serving
mirrored pulls.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The seed KS owns influxdb-seed-ssh Secret in the influxdb namespace, but
the namespace was previously created by apps/influxdb/namespace.yaml —
which dependsOn seed → seed never reconciles → circular.
Hoist the namespace into infrastructure/namespaces/namespaces.yaml
(consistent with all the other app namespaces) and drop it from the
apps/influxdb Kustomization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rolls back every file added for the Coder deployment this session —
app manifests, Flux Kustomization, namespace, CiliumNetworkPolicy,
ingress-nginx egress entry, desktop workspace template.
Flux prune will tear down the running resources (HelmRelease, CNPG
Cluster, PVCs, Ingress, TLS cert). Garage bucket data at
s3://k8s-cnpg/coder-pg and the stray Cloudflare A record for
coder.renner.uno remain as orphans.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seed Kustomization tried to create the PVC in `home-assistant` ns but
the ns lived in the `home-assistant` app Kustomization, which depends
on seed — deadlock. Matches the music-assistant pattern now: namespace
(with privileged PSA labels for hostNetwork+USB) lives in
infrastructure/namespaces, apps just consume it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces EOL Kasm VM 124 with a K8s-native platform for dynamically
provisioned desktop environments. Kasm was ruled out — its Helm chart
is Technical Preview and does not include the agent, Proxmox is being
decommissioned, and Talos refuses DinD.
HelmRelease `coder` (chart >=2.32.0 from helm.coder.com/v2), CNPG
Cluster `coder-pg` (2 instances, ceph-block 10Gi) as external DB,
SOPS-encrypted Authentik OIDC credentials, ingress at coder.renner.uno
behind letsencrypt-staging. Workspace templates follow separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Host-networked Deployment on the edge node (workload-class=pinned,
topology.kubernetes.io/zone=edge) so mDNS / uPnP / SlimProto reach
Chromecasts, DLNA renderers, and Squeezeboxes on v30iot with
native-kernel network path (lowest latency/jitter). State on a
ceph-block PVC, retained on Kustomization retirement.
Prereqs bundled in:
* rook-ceph operator: csi.pluginTolerations so the CSI node plugin
lands on the pinned edge node and MA can mount RBD.
* ingress-isolation: allow ingress-nginx egress to remote-node, since
hostNetwork pods take node identity and namespace selectors don't
match them.
* music-assistant namespace labeled pod-security enforce=privileged
(required for hostNetwork + SYS_ADMIN/DAC_READ_SEARCH + AppArmor
unconfined, matching the LXC docker-compose).
LXC 229 stays running for now; decommission once K8s side is verified.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three Flux Kustomizations chained via immich-db → immich-seed → immich:
- immich-db: CNPG Cluster using corentingiraud dual-extension image;
bootstrap.recovery from s3://k8s-cnpg/im-pg (seeded by the
migrate-immich pipeline). Barman ObjectStore + daily ScheduledBackup.
- immich-seed: library+ml-cache PVCs (RWM ceph-filesystem, expandable),
alpine Job that streams the 171 GiB tarball from s3://k8s-migration/
immich/ and writes /library/.restore-done sentinel. Prune disabled so
PVCs survive Kustomization retirement.
- immich: server + ML + redis Deployments, ConfigMap for non-secret env,
Ingress with letsencrypt-staging (dev cluster), init container that
blocks on the restore sentinel, LE-staging CA fetch for OIDC against
auth.renner.uno. DB creds via CNPG-managed im-pg-app Secret.
DB_VECTOR_EXTENSION intentionally unset — Immich >=1.133 defaults to
vchord and auto-migrates pgvecto.rs → VectorChord on first boot against
the dual-extension image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CNPG 1.29's plugin controller ONLY discovers plugin Services and
their mTLS secrets in the operator's own namespace, per
internal/controller/plugin_controller.go:
// We only consider the secrets that are installed in the
// operator namespace because plugins need to be deployed
// in the same namespace as the operator.
if obj.GetNamespace() != r.OperatorNamespace {
return nil
}
[...]
client.InNamespace(r.OperatorNamespace),
The vendored manifest installed barman-cloud into cnpg-system, which
the operator (running in database) was blind to — hence the
phaseReason "Unknown plugin: 'barman-cloud.cloudnative-pg.io'. This
may be caused by the plugin not being loaded correctly by the operator."
Rewrote the manifest to target `database` (12 namespace fields
flipped). Removed the now-orphan cnpg-system namespace definition.
Reverted the temporary --log-level=debug arg on the operator.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vendored upstream manifest from
github.com/cloudnative-pg/plugin-barman-cloud releases/download/v0.12.0/manifest.yaml.
Installs the plugin controller into cnpg-system namespace with its
ObjectStore CRD (barmancloud.cnpg.io/v1) and cert-manager-issued
self-signed client/server TLS. cnpg-system namespace added to the
namespaces manifest.
Paired ObjectStore CRs + Cluster spec wiring for ak-pg / pprl-pg
come in a follow-up commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two blockers surfaced when the Velero HelmRelease reconciled:
1. velero namespace defaulted to `baseline` PSS — node-agent
DaemonSet needs hostPath mounts for /var/lib/kubelet/{pods,plugins}
and was rejected with PodSecurity violations. Labelled the
namespace `privileged` to match the rook-ceph/crowdsec/ingress
pattern.
2. The cluster had no `snapshot.storage.k8s.io` CRDs — Rook-Ceph
CSI does not ship them. Added the piraeus snapshot-controller
chart in kube-system; it provides the VolumeSnapshot,
VolumeSnapshotContent, and VolumeSnapshotClass CRDs plus the
controller that reconciles them. Required for Velero CSI
snapshot data movement and for our rook-ceph VolumeSnapshotClass
manifest to apply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Velero via HelmRelease in a new `velero` namespace, configured
for CSI snapshot data movement to the Garage S3 endpoint on nas-1
(k8s-velero bucket). Credentials SOPS-encrypted via the repo's age
recipient; AWS plugin loaded as an initContainer; node-agent enabled
for data movement and FSB fallback; native VSL snapshots disabled in
favour of the CSI path.
Paired with a Rook-Ceph VolumeSnapshotClass for RBD (default) and
CephFS — labelled for Velero auto-discovery. Adds a `velero`
namespace definition so Flux has somewhere to put everything.
No checksum gymnastics required on Garage today, but keeps the
aws-sdk-go-v2 `checksumAlgorithm=""` workaround documented in-place
against regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the Paperless pattern: new namespace, RWX PVCs, OBC-based S3
migration bucket, trigger-backup-job that kicks a Woodpecker pipeline,
restore-data-job that streams tarballs directly from S3 into PVCs.
The Deployment uses an initContainer + sentinel file so the app never
boots on an empty /config while the SQLite tarball is still extracting,
and the restore job is idempotent (skipped when the sentinel exists).
Also extends bootstrap.sh wait-apps + e2e-test.sh to cover the new
Kustomization, OBC, TLS cert, and /healthcheck on audiobooks.k.renner.uno.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploy paperless-ngx with CNPG PostgreSQL, Redis, Gotenberg, Tika in
dedicated paperless namespace. OIDC via internal Authentik. Exposed
on paperless.k.renner.uno and docs.k.renner.uno for testing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CrowdSec agents need hostPath volumes to read container logs, which
requires a privileged PodSecurity namespace. Move from apps to
dedicated crowdsec namespace.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cilium with kubeProxyReplacement=true has native LoadBalancer support
with L2 announcements. MetalLB L2 VIPs weren't reachable because
Cilium was handling the service datapath.
- Enable l2announcements and externalIPs in Cilium HelmRelease
- Add CiliumLoadBalancerIPPool (10.2.40.64-95) and L2AnnouncementPolicy
- Remove MetalLB from infrastructure
- Remove MetalLB annotation from ingress-nginx service
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lessons from manual validation run:
- Add privileged PodSecurity labels to metallb-system, rook-ceph, ingress namespaces
- Add Cilium HelmRelease with Talos-specific settings (cgroup, kubePrism)
- Consolidate all namespace definitions in namespaces/
- Update cert-manager to use crds.enabled instead of installCRDs
- Enable all infrastructure components in main kustomization
- Add Talos secrets to .gitignore (controlplane.yaml, talosconfig)
- Update node patches with correct install disk (vda) and zone labels
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
claude-bot
k.rennerandClaude Opus 4.8