Compare commits

..
Author SHA1 Message Date
hermes-bot 766a790ee1 fix(crowdsec): sign the LAPI/agent certs with a CA that outlives them
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
The LAPI has been in CrashLoopBackOff since the restart #1394 triggered
(10:08Z). Its own watcher login fails:
  tls: failed to verify certificate: x509: certificate has expired or is
  not yet valid: current time 2026-10-20T10:13:44Z is after
  2026-10-20T06:24:37Z
Rolling back to the previous ReplicaSet fails the same way, so any LAPI
restart since this morning would have hit it.

Cause: the chart's CA Certificate sets no duration, so cert-manager's 90d
default applies, the same lifetime as the leaves (tls.certManager.duration
90d, renewBefore 30d). All of them renew together. At 2026-09-20 06:24:37
the leaves were re-signed by the CA issued 2026-07-22 06:24:37 while the
CA itself rotated to a new key. That earlier CA expired today at 06:24:37,
so every leaf now chains to an expired CA. The LAPI pod running since
2026-10-01 kept working with the certs already loaded, until the restart.
The same skew is what made Traefik reject the LAPI cert (500) since
2026-09-20.

Fix, no chart fork: our own CA in tls-ca.yaml (self-signed Issuer, 10y CA
Certificate renewed a year before expiry, CA Issuer), handed to the chart
via tls.certManager.issuerRef. The chart then stops rendering its own
CA/issuers. cert-manager re-issues every leaf because the Secret's issuer
annotation no longer matches the Certificate's issuerRef
(SecretIssuerAnnotationsMismatch). Leaves renew every 60d under a CA that
lives 10y, so the leaves and the CA no longer renew together.

Reloader restarts LAPI (crowdsec-lapi-tls, crowdsec-agent-tls) and agents
(crowdsec-agent-tls) when the certs change; both read them at start only.
The ServersTransport already trusts crowdsec-lapi-tls's ca.crt (#1394), so
it follows the new CA.
2026-10-20 10:15:31 +00:00
k.renner bca807c2c1 Merge pull request 'fix(crowdsec): trust the CA that signed the LAPI cert, not the current root' (#1394) from fix/crowdsec-lapi-transport-ca into main
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: #1394
2026-10-20 10:06:46 +00:00
k.renner a4d60ad689 Merge pull request 'fix(matrix): disable service links for LiveKit and lk-jwt-service' (#1395) from fix/livekit-service-links into main
Reviewed-on: #1395
2026-10-20 10:06:08 +00:00
hermes-bot f96cef137d fix(matrix): disable service links for LiveKit and lk-jwt-service
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
LiveKit crashlooped after #1393:
  could not parse "tcp://10.107.186.157:7880" as uint value from
  environment variable "LIVEKIT_PORT" for flag port

The Service named livekit makes the kubelet inject Docker-style link
variables (LIVEKIT_PORT, LIVEKIT_SERVICE_HOST, ...) into every pod in the
namespace. LiveKit reads LIVEKIT_* as its own config; lk-jwt-service reads
LIVEKIT_* too. enableServiceLinks: false on both pods removes them.
2026-10-20 10:04:28 +00:00
hermes-bot 34a8f3d2f3 fix(crowdsec): trust the CA that signed the LAPI cert, not the current root
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
crowdsec.renner.uno has answered 500 on every path since 2026-09-20.
Traefik logs, every request:
  tls: failed to verify certificate: x509: certificate signed by unknown
  authority (possibly because of "x509: ECDSA verification failure" while
  trying to verify candidate authority certificate "crowdsec-ca")

Cause: the chart's CA (crowdsec-ca, no duration set, so cert-manager's 90d
default) and its leaves (tls.certManager.duration 90d, renewBefore 30d) were
created together and so renew at the same instant. cert-manager logs show
crowdsec-ca, crowdsec-lapi and crowdsec-agent all triggered "Renewing" at
2026-09-20T06:24:37Z. The CA came back with a new ECDSA key, while the
leaves issued in the same second were signed by the previous one. The
ServersTransport trusted crowdsec-root-secret, so it holds the new CA and
rejects the LAPI cert. Same subject, different key, hence "ECDSA
verification failure".

Fix: point the ServersTransport at the ca.crt of crowdsec-lapi-tls. For a
CA issuer, cert-manager stores the signing CA there, so it matches the
served cert whichever key the root holds. Traefik reads ca.crt from a
rootCAs secret (Traefik v3.7 docs).

Reloader also restarts the LAPI when crowdsec-lapi-tls changes. The LAPI
reads its cert at start; without a restart a renewal would update ca.crt
while the old leaf is still served, and the mismatch would come back.

Impact while broken: the edge firewall bouncers on proxy-1/proxy-2 got no
decisions from this endpoint.
2026-10-20 10:04:18 +00:00
security-bot 783bb70a0e chore(image-automation): bump gitea.renner.uno/gitea_admin/kuma-sync:2026-10-20-0933
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-10-20 09:56:44 +00:00
k.renner de5ca66466 Merge pull request 'feat(matrix): Element Call (audio and video) via LiveKit + lk-jwt-service' (#1393) from feat/matrix-element-call into main
Reviewed-on: #1393
2026-10-20 09:56:13 +00:00
6 changed files with 55 additions and 10 deletions
+1 -1
View File
@@ -30,7 +30,7 @@ spec:
type: RuntimeDefault
containers:
- name: controller
image: gitea.renner.uno/gitea_admin/kuma-sync:2026-10-19-0857 # {"$imagepolicy": "flux-system:kuma-sync"}
image: gitea.renner.uno/gitea_admin/kuma-sync:2026-10-20-0933 # {"$imagepolicy": "flux-system:kuma-sync"}
env:
- name: KUMA_URL
value: https://kuma.renner.uno
+4
View File
@@ -114,6 +114,8 @@ spec:
app.kubernetes.io/part-of: matrix
spec:
automountServiceAccountToken: false
# Service links would inject LIVEKIT_PORT=tcp://... (Service livekit), which LiveKit and lk-jwt-service read as config.
enableServiceLinks: false
securityContext:
runAsNonRoot: true
runAsUser: 65534
@@ -212,6 +214,8 @@ spec:
app.kubernetes.io/part-of: matrix
spec:
automountServiceAccountToken: false
# Service links would inject LIVEKIT_PORT=tcp://... (Service livekit), which LiveKit and lk-jwt-service read as config.
enableServiceLinks: false
securityContext:
runAsNonRoot: true
runAsUser: 65534
@@ -93,10 +93,10 @@ spec:
# self-heals with no PVC and no intervention.
#
# cert-manager (in `infrastructure`, which the `crowdsec` Kustomization
# depends on) provisions a fully self-contained self-signed CA (selfSigned
# ClusterIssuer -> CA cert -> CA Issuer -> lapi/agent/bouncer leaf certs);
# empty issuerRef => the chart mints that CA itself, so nothing external is
# required.
# depends on) signs the lapi/agent/bouncer leaf certs with the CA in
# tls-ca.yaml. Not the chart's own CA: that one has cert-manager's 90d
# default, the same lifetime as the leaves, so the leaves chain to a CA
# that expires 30 days after they were issued.
#
# NOTE: enabling this makes the LAPI serve HTTPS-only. The Traefik ingress
# bouncer is currently INERT (detached, see post-crds/traefik-crowdsec-middleware.yaml)
@@ -107,8 +107,9 @@ spec:
caBundle: true
certManager:
enabled: true
# Empty => chart provisions its own self-signed CA (self-contained).
issuerRef: {}
issuerRef:
name: crowdsec-tls-ca-issuer
kind: Issuer
# Chart 0.24.0 bug: the agent/bouncer/appsec Certificate templates emit
# `secretTemplate:` unconditionally while its child blocks are guarded,
# so with nothing set it renders as null and cert-manager's webhook
@@ -160,7 +161,8 @@ spec:
# crowdsec-edge-bouncer-key rotates by hand together with
# ansible/plays/install-netbird-proxy.yml; it still needs to be listed
# here, because the LAPI side of that rotation is this env var.
secret.reloader.stakater.com/reload: crowdsec-bouncer-key,crowdsec-edge-bouncer-key
# crowdsec-lapi-tls, crowdsec-agent-tls: certs are read at start only.
secret.reloader.stakater.com/reload: crowdsec-bouncer-key,crowdsec-edge-bouncer-key,crowdsec-lapi-tls,crowdsec-agent-tls
persistentVolume:
data:
# Kept OFF deliberately. The #337 attempt to set this `true` (persist
@@ -203,6 +205,9 @@ spec:
key: key
agent:
podAnnotations:
# The agent reads its client cert at start only.
secret.reloader.stakater.com/reload: crowdsec-agent-tls
# Same reasoning as lapi.resources above, multiplied by three: this is a
# DaemonSet, so the chart's 500m default reserves half a core on every
# node to run a process that measures 1-2m.
@@ -2,6 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- lapi-ingress.yaml
- tls-ca.yaml
- helmrelease.yaml
- crowdsec-bouncer-key-externalsecret.yaml
- crowdsec-edge-bouncer-key-externalsecret.yaml
@@ -0,0 +1,35 @@
# CA for the chart's LAPI/agent/bouncer certs (tls.certManager.issuerRef); outlives them by design.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: crowdsec-tls-selfsigned
namespace: crowdsec
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: crowdsec-tls-ca
namespace: crowdsec
spec:
isCA: true
commonName: crowdsec-tls-ca
secretName: crowdsec-tls-ca
duration: 87600h
renewBefore: 8760h
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: crowdsec-tls-selfsigned
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: crowdsec-tls-ca-issuer
namespace: crowdsec
spec:
ca:
secretName: crowdsec-tls-ca
@@ -1,5 +1,5 @@
# traefik.io CRD consumer for the public LAPI endpoint (crowdsec/lapi-ingress.yaml
# holds the Ingress). The CA it names is the chart's self-signed LAPI CA.
# holds the Ingress). Trusts ca.crt of the LAPI's own cert secret: the CA that signed it.
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
@@ -8,4 +8,4 @@ metadata:
spec:
serverName: crowdsec-service.crowdsec.svc.cluster.local
rootCAs:
- secret: crowdsec-root-secret
- secret: crowdsec-lapi-tls