feat(matrix): Element Call (audio and video) via LiveKit + lk-jwt-service

Element X reported MISSING_MATRIX_RTC_TRANSPORT: matrix.renner.uno had no
MatrixRTC backend (no SFU, /rtc/transports 404, msc4143 off).

Adds the backend from the Element Call self-hosting guide and the
lk-jwt-service README:

- LiveKit SFU (rtc.yaml), signalling at wss://rtc.matrix.renner.uno.
  Media: ICE straight to the pod IPs (UDP 7882 mux, TCP 7881), which the
  LAN reaches through the BGP-routed pod CIDR, dual-stack. Clients that
  cannot reach the pod (mobile data, NetBird-only) use the embedded TURN
  server over TLS on turn.matrix.renner.uno:443. That rides the existing
  public path: NetBird proxy TLS passthrough -> Traefik, where an
  IngressRouteTCP terminates TLS (turn.external_tls) and hands plain TCP to
  LiveKit. The proxy only does TLS on :443, so TURN/TLS is the one transport
  that works from the internet without opening router ports.
- lk-jwt-service as a Synapse appservice (MSC4502 membership look-ups,
  MSC4512 proxy of /rtc/livekit to it), plus its legacy client endpoints
  under rtc.matrix.renner.uno/livekit/jwt: Element Call 0.28 / Element Web
  1.12.32 still use livekit_service_url.
- The legacy endpoints verify OpenID tokens at
  <server>/_matrix/federation/v1/openid/userinfo, found through
  /.well-known/matrix/server (fallback :8448). Synapse therefore serves the
  server well-known and the `openid` listener resource, and the Ingress
  routes exactly those two paths. No federation resource is served and
  federation_domain_whitelist stays empty; federation remains off.
- Synapse: msc4143/4222/4502/4512, matrix_rtc transport (url +
  livekit_service_url), rtc_foci in the client well-known for older
  embedded Element Call builds, delayed events (24h) and the guide's rate
  limits for key sharing and membership heartbeats.
- Secrets: one Password generator (LiveKit API secret, appservice tokens),
  generated once in-cluster like the former matrix-bridge tokens; Synapse,
  LiveKit and lk-jwt-service read the same Secret, so nothing can drift and
  nothing needs seeding in OpenBao.
- CNPs: world -> LiveKit 7881/tcp + 7882/udp and LiveKit -> world UDP
  (ICE checks/media); lk-jwt-service -> Traefik 8443 (matrix.renner.uno and
  rtc.matrix.renner.uno resolve to the Traefik VIP in-cluster).
- embedded TURN denies private peers by default; the relay targets the
  SFU's own pod IP, so 10.0.0.0/8 and fc00::/7 are allowed (LiveKit already
  limits embedded-TURN relays to its own media ports).
- Kuma: DNS + reachability for rtc. and turn.matrix.renner.uno.

kubernetes/apps/matrix/ingress.yaml

@@ -100,6 +100,21 @@ spec:
                 name: synapse
                 port:
                   number: 8008
+          # lk-jwt-service resolves the server via this file and verifies OpenID tokens at the path below.
+          - path: /.well-known/matrix/server
+            pathType: Exact
+            backend:
+              service:
+                name: synapse
+                port:
+                  number: 8008
+          - path: /_matrix/federation/v1/openid/userinfo
+            pathType: Exact
+            backend:
+              service:
+                name: synapse
+                port:
+                  number: 8008
 ---
 apiVersion: networking.k8s.io/v1
 kind: Ingress

kubernetes/apps/matrix/kumamonitors.yaml

@@ -272,3 +272,90 @@ spec:
     expression: "default_server_config.`m.homeserver`.base_url"
     operator: "=="
     expected: "https://matrix.renner.uno"
+---
+apiVersion: uptime.renner.uno/v1alpha1
+kind: KumaMonitor
+metadata:
+  name: rtc-dns
+  namespace: matrix
+spec:
+  type: dns
+  displayName: MatrixRTC DNS
+  parentRef:
+    name: matrix
+  interval: 60
+  retryInterval: 60
+  maxRetries: 1
+  dns:
+    # cluster services have no own record; they resolve through the *.renner.uno wildcard
+    hostname: rtc.matrix.renner.uno
+    resolveType: CNAME
+    resolveServer: "2606:4700:4700::1111,1.1.1.1"
+    conditions:
+      - operator: equals
+        value: proxy.renner.uno
+---
+apiVersion: uptime.renner.uno/v1alpha1
+kind: KumaMonitor
+metadata:
+  name: rtc-sfu
+  namespace: matrix
+spec:
+  type: keyword
+  displayName: LiveKit SFU
+  parentRef:
+    name: matrix
+  interval: 60
+  retryInterval: 60
+  maxRetries: 1
+  timeout: 10
+  http:
+    # LiveKit's health answer on its signalling port
+    url: https://rtc.matrix.renner.uno/
+    acceptedStatusCodes: ["200-299"]
+    expiryNotification: true
+  keyword:
+    keyword: "OK"
+---
+apiVersion: uptime.renner.uno/v1alpha1
+kind: KumaMonitor
+metadata:
+  name: rtc-jwt
+  namespace: matrix
+spec:
+  type: http
+  displayName: MatrixRTC Authorization Service
+  parentRef:
+    name: matrix
+  interval: 60
+  retryInterval: 60
+  maxRetries: 1
+  timeout: 10
+  http:
+    # reachable: any answer from the backend below 5xx (an unauthenticated GET is rejected with 4xx)
+    url: https://rtc.matrix.renner.uno/livekit/jwt/sfu/get
+    acceptedStatusCodes: ["200-499"]
+    maxRedirects: 0
+    expiryNotification: true
+---
+apiVersion: uptime.renner.uno/v1alpha1
+kind: KumaMonitor
+metadata:
+  name: turn-dns
+  namespace: matrix
+spec:
+  type: dns
+  displayName: TURN DNS
+  parentRef:
+    name: matrix
+  interval: 60
+  retryInterval: 60
+  maxRetries: 1
+  dns:
+    # cluster services have no own record; they resolve through the *.renner.uno wildcard
+    hostname: turn.matrix.renner.uno
+    resolveType: CNAME
+    resolveServer: "2606:4700:4700::1111,1.1.1.1"
+    conditions:
+      - operator: equals
+        value: proxy.renner.uno

kubernetes/apps/matrix/kustomization.yaml

@@ -5,6 +5,7 @@ resources:
   - synapse.yaml
   - mas.yaml
   - element.yaml
+  - rtc.yaml
   - ingress.yaml
   # mx-pg CNPG Postgres + offsite Barman backup. The media store and signing
   # key are on synapse-data, backed up by Velero (matrix-daily).

kubernetes/apps/matrix/rtc.yaml

@@ -0,0 +1,387 @@
+# MatrixRTC backend for Element Call: LiveKit SFU + lk-jwt-service.
+# Generated once in-cluster; Synapse, LiveKit and lk-jwt-service read the same Secret.
+apiVersion: generators.external-secrets.io/v1alpha1
+kind: Password
+metadata:
+  name: matrix-rtc
+  namespace: matrix
+spec:
+  length: 64
+  digits: 16
+  symbols: 0
+  allowRepeat: true
+  noUpper: false
+  secretKeys:
+    - livekit_secret
+    - as_token
+    - hs_token
+---
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: matrix-rtc
+  namespace: matrix
+spec:
+  refreshPolicy: CreatedOnce
+  target:
+    name: matrix-rtc
+    template:
+      engineVersion: v2
+      data:
+        livekit_secret: "{{ .livekit_secret }}"
+        # LIVEKIT_KEYS format "<key>: <secret>"; the key name is the fixed id matrix-rtc.
+        livekit_keys: "matrix-rtc: {{ .livekit_secret }}"
+        # Appservice registration: membership look-ups and the /rtc/livekit proxy only, no event traffic.
+        registration.yaml: |
+          id: lk-jwt-service
+          url: null
+          as_token: "{{ .as_token }}"
+          hs_token: "{{ .hs_token }}"
+          sender_localpart: _lk_jwt_service
+          rate_limited: false
+          namespaces:
+            users:
+              - exclusive: false
+                regex: ".*"
+            aliases: []
+            rooms: []
+          io.element.msc4502.scopes: ["urn:matrix:client:io.element.msc4502:rooms:is_joined"]
+          io.element.msc4512.proxy_prefix: "rtc/livekit"
+          io.element.msc4512.proxy_url: "http://lk-jwt-service:8080"
+  dataFrom:
+    - sourceRef:
+        generatorRef:
+          apiVersion: generators.external-secrets.io/v1alpha1
+          kind: Password
+          name: matrix-rtc
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: livekit-config
+  namespace: matrix
+data:
+  livekit.yaml: |
+    port: 7880
+    rtc:
+      # Host candidates are the pod IPs (dual-stack, BGP-routed to the LAN).
+      use_external_ip: false
+      tcp_port: 7881
+      udp_port: 7882
+    turn:
+      # TURN/TLS on turn.matrix.renner.uno:443; Traefik terminates TLS (IngressRouteTCP livekit-turn).
+      enabled: true
+      domain: turn.matrix.renner.uno
+      tls_port: 443
+      external_tls: true
+      # The relay targets the SFU's own pod IP; LiveKit limits embedded-TURN relays to its own media ports.
+      allow_restricted_peer_cidrs:
+        - 10.0.0.0/8
+        - fc00::/7
+    room:
+      # lk-jwt-service creates rooms for matrix.renner.uno users only.
+      auto_create: false
+    webhook:
+      api_key: matrix-rtc
+      urls:
+        - http://lk-jwt-service:8080/sfu_webhook
+    logging:
+      level: info
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: livekit
+  namespace: matrix
+  annotations:
+    configmap.reloader.stakater.com/reload: livekit-config
+    secret.reloader.stakater.com/reload: matrix-rtc
+  labels:
+    app.kubernetes.io/name: livekit
+    app.kubernetes.io/part-of: matrix
+spec:
+  replicas: 1
+  # One SFU: two pods would split a call's participants.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: livekit
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: livekit
+        app.kubernetes.io/part-of: matrix
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: livekit
+          image: livekit/livekit-server:v1.13.10@sha256:89ca9ce0791e18d6e03bf6e2c49ceb27e1c754eae8bd50c989386c1a5963791a
+          args:
+            - --config
+            - /config/livekit.yaml
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          env:
+            - name: LIVEKIT_KEYS
+              valueFrom:
+                secretKeyRef:
+                  name: matrix-rtc
+                  key: livekit_keys
+          ports:
+            - name: http
+              containerPort: 7880
+            - name: ice-tcp
+              containerPort: 7881
+            - name: ice-udp
+              containerPort: 7882
+              protocol: UDP
+            - name: turn
+              containerPort: 443
+          resources:
+            requests:
+              cpu: 100m
+              memory: 128Mi
+            limits:
+              memory: 1Gi
+          readinessProbe:
+            httpGet:
+              path: /
+              port: http
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 5
+            failureThreshold: 4
+          volumeMounts:
+            - name: config
+              mountPath: /config
+              readOnly: true
+      volumes:
+        - name: config
+          configMap:
+            name: livekit-config
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: livekit
+  namespace: matrix
+spec:
+  selector:
+    app.kubernetes.io/name: livekit
+  ports:
+    - name: http
+      port: 7880
+      targetPort: http
+    - name: turn
+      port: 443
+      targetPort: turn
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: lk-jwt-service
+  namespace: matrix
+  annotations:
+    secret.reloader.stakater.com/reload: matrix-rtc
+  labels:
+    app.kubernetes.io/name: lk-jwt-service
+    app.kubernetes.io/part-of: matrix
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: lk-jwt-service
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: lk-jwt-service
+        app.kubernetes.io/part-of: matrix
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: lk-jwt-service
+          image: ghcr.io/element-hq/lk-jwt-service:0.9.0@sha256:d210fd778ba38b7e42e6fc64e8a24a7d66ccc3b938fb20d73bd713224af2dfea
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          env:
+            # Must equal matrix_rtc.transports[].url in synapse.yaml.
+            - name: LIVEKIT_URL
+              value: wss://rtc.matrix.renner.uno
+            - name: LIVEKIT_KEY
+              value: matrix-rtc
+            - name: LIVEKIT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: matrix-rtc
+                  key: livekit_secret
+            - name: LIVEKIT_FULL_ACCESS_HOMESERVERS
+              value: matrix.renner.uno
+            # Federation is off: no remote homeserver publishes on this SFU.
+            - name: LIVEKIT_REMOTE_CAN_PUBLISH_HOMESERVERS
+              value: ""
+            - name: LIVEKIT_AS_REGISTRATION_FILE
+              value: /registration/registration.yaml
+            - name: LIVEKIT_HS_SERVER_NAME
+              value: matrix.renner.uno
+            - name: LIVEKIT_CS_API_URL_OVERRIDES
+              value: matrix.renner.uno=http://synapse:8008
+            - name: LIVEKIT_SANITY_CHECK_INTERVAL_SECONDS
+              value: "60"
+          ports:
+            - name: http
+              containerPort: 8080
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 256Mi
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: http
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 5
+            failureThreshold: 4
+          volumeMounts:
+            - name: registration
+              mountPath: /registration
+              readOnly: true
+      volumes:
+        - name: registration
+          secret:
+            secretName: matrix-rtc
+            items:
+              - key: registration.yaml
+                path: registration.yaml
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: lk-jwt-service
+  namespace: matrix
+spec:
+  selector:
+    app.kubernetes.io/name: lk-jwt-service
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http
+---
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: lk-jwt-strip
+  namespace: matrix
+spec:
+  stripPrefix:
+    prefixes:
+      - /livekit/jwt
+---
+# turn.matrix.renner.uno is listed here to be published and get a certificate; IngressRouteTCP livekit-turn serves it.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: matrix-rtc
+  namespace: matrix
+  labels:
+    dns.renner.uno/public: "true"
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+    traefik.ingress.kubernetes.io/router.middlewares: matrix-lk-jwt-strip@kubernetescrd
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - rtc.matrix.renner.uno
+      secretName: rtc-tls
+    - hosts:
+        - turn.matrix.renner.uno
+      secretName: turn-tls
+  rules:
+    - host: rtc.matrix.renner.uno
+      http:
+        paths:
+          # Client endpoints of lk-jwt-service (livekit_service_url); the rest of the service stays in-cluster.
+          - path: /livekit/jwt/sfu/get
+            pathType: Exact
+            backend:
+              service:
+                name: lk-jwt-service
+                port:
+                  number: 8080
+          - path: /livekit/jwt/get_token
+            pathType: Exact
+            backend:
+              service:
+                name: lk-jwt-service
+                port:
+                  number: 8080
+          - path: /livekit/jwt/delegate_delayed_leave
+            pathType: Exact
+            backend:
+              service:
+                name: lk-jwt-service
+                port:
+                  number: 8080
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: livekit
+                port:
+                  number: 7880
+    - host: turn.matrix.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: livekit
+                port:
+                  number: 7880
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRouteTCP
+metadata:
+  name: livekit-turn
+  namespace: matrix
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: HostSNI(`turn.matrix.renner.uno`)
+      services:
+        - name: livekit
+          port: 443
+  tls:
+    secretName: turn-tls

kubernetes/apps/matrix/synapse.yaml

@@ -19,8 +19,15 @@ data:
     server_name: matrix.renner.uno
     public_baseurl: https://matrix.renner.uno/
     # Lets Element X and other clients discover the homeserver from the
-    # server name. No server well-known: there is no federation to point at.
+    # server name.
     serve_client_wellknown: true
+    # Delegates to :443 so lk-jwt-service finds /_matrix/federation/v1/openid/userinfo (openid resource below).
+    serve_server_wellknown: true
+    # Element Call builds that still read the MatrixRTC transport from the well-known.
+    extra_well_known_client_content:
+      org.matrix.msc4143.rtc_foci:
+        - type: livekit
+          livekit_service_url: https://rtc.matrix.renner.uno/livekit/jwt
     pid_file: /tmp/homeserver.pid
     report_stats: false
 
@@ -34,7 +41,8 @@ data:
         bind_addresses: ["::"]
         x_forwarded: true
         resources:
-          - names: [client]
+          # openid: only the OpenID userinfo endpoint lk-jwt-service verifies call tokens with; no federation.
+          - names: [client, openid]
             compress: false
 
     # The password comes from PGPASSWORD (libpq reads it), so no credential
@@ -73,6 +81,30 @@ data:
     experimental_features:
       msc4108_enabled: true
       msc4388_mode: open
+      # MatrixRTC (Element Call): transports endpoint, sync state_after,
+      # appservice membership look-ups and the /rtc/livekit proxy to lk-jwt-service.
+      msc4143_enabled: true
+      msc4222_enabled: true
+      msc4502_enabled: true
+      msc4512_enabled: true
+
+    # MatrixRTC transport (rtc.yaml). url must equal LIVEKIT_URL of lk-jwt-service.
+    matrix_rtc:
+      transports:
+        - type: livekit
+          url: wss://rtc.matrix.renner.uno
+          livekit_service_url: https://rtc.matrix.renner.uno/livekit/jwt
+    app_service_config_files:
+      - /appservices/lk-jwt-service/registration.yaml
+    # Call membership leaves via delayed events (MSC4140).
+    max_event_delay_duration: 24h
+    # Element Call self-hosting guide values: E2EE key sharing bursts and 5 s membership heartbeats.
+    rc_message:
+      per_second: 0.5
+      burst_count: 30
+    rc_delayed_event_mgmt:
+      per_second: 1
+      burst_count: 20
 
     # Federation off.
     federation_domain_whitelist: []
@@ -115,7 +147,7 @@ metadata:
   namespace: matrix
   annotations:
     configmap.reloader.stakater.com/reload: synapse-config,mas-config
-    secret.reloader.stakater.com/reload: synapse-secrets
+    secret.reloader.stakater.com/reload: synapse-secrets,matrix-rtc
   labels:
     app.kubernetes.io/name: synapse
     app.kubernetes.io/part-of: matrix
@@ -263,6 +295,9 @@ spec:
               mountPath: /data
             - name: tmp
               mountPath: /tmp
+            - name: appservice-lk-jwt
+              mountPath: /appservices/lk-jwt-service
+              readOnly: true
       volumes:
         - name: config
           configMap:
@@ -295,6 +330,12 @@ spec:
             claimName: synapse-data
         - name: tmp
           emptyDir: {}
+        - name: appservice-lk-jwt
+          secret:
+            secretName: matrix-rtc
+            items:
+              - key: registration.yaml
+                path: registration.yaml
 ---
 apiVersion: v1
 kind: Service

kubernetes/infrastructure/network-policies/matrix-ns.yaml

@@ -137,3 +137,53 @@ spec:
         - ports:
             - port: "8443"
               protocol: TCP
+---
+# LiveKit media: ICE from clients straight to the pod IPs (LAN via the BGP-routed pod CIDR).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: livekit-media
+  namespace: matrix
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: livekit
+  ingress:
+    - fromEntities:
+        - world
+      toPorts:
+        - ports:
+            - port: "7881"
+              protocol: TCP
+            - port: "7882"
+              protocol: UDP
+  egress:
+    # ICE connectivity checks and media to client candidates.
+    - toEntities:
+        - world
+      toPorts:
+        - ports:
+            - port: "1"
+              endPort: 65535
+              protocol: UDP
+---
+# lk-jwt-service → matrix.renner.uno (server well-known, OpenID userinfo) and
+# rtc.matrix.renner.uno (LiveKit room API); both resolve to the Traefik VIP.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: lk-jwt-service-egress
+  namespace: matrix
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: lk-jwt-service
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8443"
+              protocol: TCP