fix(crowdsec): move the LAPI backend annotations onto the Service

Both bouncers reach the LAPI and pull every 10s, and every pull gets a 400:

  2a03:4000:28:f6e:...:9a29 "GET /v1/decisions/stream?..." 400 48
  2a03:4000:4b:c1e:...:8f85 "GET /v1/decisions/stream?..." 400 48

The body says what it is: "Client sent an HTTP request to an HTTPS server."
Traefik was talking plaintext to a backend that serves TLS only.

serversscheme and serverstransport were on the Ingress. Under the Kubernetes
Ingress provider `traefik.ingress.kubernetes.io/service.*` are SERVICE
annotations; on an Ingress they are silently ignored. Nothing warns, and the
failure surfaces as a 400 that reads like a malformed request rather than a
scheme mismatch. They move to the chart's Service via lapi.service.annotations.

With tls.enabled the LAPI serves HTTPS on 8080 and offers no plaintext port,
so https is mandatory here rather than hardening.

Also switches the ServersTransport from rootCAsSecrets to rootCAs, which is
what Traefik has been warning about on every reconcile:

  RootCAsSecrets option is deprecated, please use the RootCA option instead.

Two of my own diagnostics were wrong along the way and both were measurement
artifacts. "Traefik sees no requests" came from a Loki query whose limit was
consumed by those deprecation warnings. And the source address that justified
dropping the IPAllowList was measured on the watchdog, which curls 127.0.0.1;
the bouncer resolves the public name and arrives on the proxy's public
address, as the log lines above show.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/crowdsec-edge-bouncer/tasks/main.yml

@@ -43,25 +43,35 @@
   become: false
   run_once: true
 
+# Only the download had to move to the controller — the extraction stays on
+# the host. The ci-ansible image ships a tar that ansible rejects ("detected as
+# tar type None. GNU tar required"), while both host distributions have GNU
+# tar. Shipping the archive rather than the unpacked binary also keeps the
+# artefact that the checksum was verified against intact all the way to the
+# host.
+- name: Copy the bouncer archive to the host
+  ansible.builtin.copy:
+    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    mode: "0644"
+
 # Only the binary. The tarball also carries install.sh / upgrade.sh /
 # uninstall.sh, which manage systemd and config themselves — that is this
 # role's job, and letting them run would leave two owners for the same files.
-- name: Unpack the bouncer binary on the controller
+- name: Unpack the bouncer binary
   ansible.builtin.unarchive:
-    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
-    dest: /tmp
+    src: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /opt
     remote_src: true
     include:
       - crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
-    creates: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
-  delegate_to: localhost
-  become: false
-  run_once: true
+    creates: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
 
 - name: Install the bouncer binary
   ansible.builtin.copy:
-    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
+    src: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
     dest: /usr/local/bin/crowdsec-firewall-bouncer
+    remote_src: true
     mode: "0755"
   notify: Restart crowdsec firewall bouncer
 

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -108,6 +108,24 @@ spec:
 
     lapi:
       replicas: 1
+      service:
+        # Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
+        #
+        # These belong on the SERVICE, not on the Ingress: under the Kubernetes
+        # Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
+        # annotations, and on an Ingress they are silently ignored. The symptom
+        # is Traefik speaking plaintext to a TLS-only backend, which the LAPI
+        # answers with "Client sent an HTTP request to an HTTPS server." and a
+        # 400 — an error that looks like a malformed request rather than a
+        # scheme mismatch.
+        #
+        # With tls.enabled the LAPI serves HTTPS on 8080 and offers no
+        # plaintext port, so https here is mandatory rather than a hardening
+        # choice. The ServersTransport carries the chart's own CA, since that
+        # certificate is self-signed.
+        annotations:
+          traefik.ingress.kubernetes.io/service.serversscheme: https
+          traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
       # Far below the chart default, which reserves half a core per LAPI for a
       # workload that draws single-digit millicores — enough to leave the
       # cluster unschedulable with one node out. Only the reservation shrinks;

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -44,8 +44,8 @@ metadata:
   namespace: crowdsec
 spec:
   serverName: crowdsec-service.crowdsec.svc.cluster.local
-  rootCAsSecrets:
-    - crowdsec-root-secret
+  rootCAs:
+    - secret: crowdsec-root-secret
 ---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
@@ -60,8 +60,12 @@ metadata:
     cert-manager.io/cluster-issuer: letsencrypt-cluster
     # NOT behind the Authentik gate: the client is a daemon with an API key,
     # not a browser that can complete an OIDC flow.
-    traefik.ingress.kubernetes.io/service.serversscheme: https
-    traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
+    #
+    # The backend scheme and ServersTransport are NOT set here. Under the
+    # Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
+    # are SERVICE annotations — on an Ingress they are silently ignored, and
+    # Traefik then talks plaintext to a TLS-only backend. They live on the
+    # chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
 spec:
   ingressClassName: traefik
   rules: