kubernetes/talos/patches/cilium-cni.yaml (the live Talos bootstrap inline manifest) and its standalone copy kubernetes/talos/cilium.yaml embedded Secret/cilium-ca (RSA ca.key) and Secret/hubble-server-certs (RSA tls.key) as plaintext base64 — rendered Helm output from hubble.tls.auto.method=helm.
This PR switches to hubble.tls.auto.method=cronJob: the two key-bearing Secrets are replaced by the certgen ServiceAccount / Role / RoleBinding / Job / CronJob. The Job generates (and the CronJob renews) Secret/cilium-ca and issues Secret/hubble-server-certs in-cluster at install. No private key material is committed.
Derivation / verification
helm template cilium/cilium --version 1.19.2 rendered both ways (method=helm vs method=cronJob); the only delta is exactly these 7 resources (2 secrets removed, 5 certgen added). ConfigMap and cilium-agent DaemonSet are byte-identical between methods, so the change is applied surgically and the rest of each manifest is untouched.
The agent hubble-tls volume is already optional:true, so cilium-agent starts before certgen has run and picks up the cert once issued (bootstrap ordering safe).
certgen cert CN/SAN *.default.hubble-grpc.cilium.io matches the cluster-name (default) in the rendered ConfigMap.
Both files and the inline contents: block scalar validated as multi-doc YAML; no committed private keys remain.
The keys already leaked into git history, so this PR alone is not sufficient:
Rotate the live cilium-ca + Hubble certs (treat committed keys as compromised).
Scrub them from git history after rotation.
Exact procedure + what is needed from the owner is posted on issue #188.
Note: standalone cilium.yaml has drifted from cilium-cni.yaml (ipv6/BGP differ) — flagged on the issue for separate reconciliation.
Fixes #188 (code half).
## What
`kubernetes/talos/patches/cilium-cni.yaml` (the live Talos bootstrap inline manifest) and its standalone copy `kubernetes/talos/cilium.yaml` embedded `Secret/cilium-ca` (RSA `ca.key`) and `Secret/hubble-server-certs` (RSA `tls.key`) as plaintext base64 — rendered Helm output from `hubble.tls.auto.method=helm`.
This PR switches to `hubble.tls.auto.method=cronJob`: the two key-bearing Secrets are replaced by the certgen ServiceAccount / Role / RoleBinding / Job / CronJob. The Job generates (and the CronJob renews) `Secret/cilium-ca` and issues `Secret/hubble-server-certs` in-cluster at install. No private key material is committed.
## Derivation / verification
- `helm template cilium/cilium --version 1.19.2` rendered both ways (method=helm vs method=cronJob); the only delta is exactly these 7 resources (2 secrets removed, 5 certgen added). ConfigMap and cilium-agent DaemonSet are byte-identical between methods, so the change is applied surgically and the rest of each manifest is untouched.
- The agent hubble-tls volume is already optional:true, so cilium-agent starts before certgen has run and picks up the cert once issued (bootstrap ordering safe).
- certgen cert CN/SAN *.default.hubble-grpc.cilium.io matches the cluster-name (default) in the rendered ConfigMap.
- Both files and the inline contents: block scalar validated as multi-doc YAML; no committed private keys remain.
## Not in this PR (operator tasks, tracked on #188)
The keys already leaked into git history, so this PR alone is not sufficient:
1. Rotate the live cilium-ca + Hubble certs (treat committed keys as compromised).
2. Scrub them from git history after rotation.
Exact procedure + what is needed from the owner is posted on issue #188.
Note: standalone cilium.yaml has drifted from cilium-cni.yaml (ipv6/BGP differ) — flagged on the issue for separate reconciliation.
The Talos bootstrap inline manifest (patches/cilium-cni.yaml) and its
standalone copy (cilium.yaml) embedded Secret/cilium-ca (RSA ca.key) and
Secret/hubble-server-certs (RSA tls.key) as plaintext base64 — rendered
Helm output from hubble.tls.auto.method=helm. Anyone with repo/history
read access could mint certs under the Cilium CA and impersonate/decrypt
the Hubble mTLS channel.
Switch to hubble.tls.auto.method=cronJob: replace the two committed-key
Secrets with the certgen ServiceAccount/Role/RoleBinding/Job/CronJob.
The Job generates (and the CronJob renews) Secret cilium-ca and issues
Secret/hubble-server-certs in-cluster at install. No private key material
is committed. Verified with `helm template cilium/cilium --version 1.19.2`:
the only delta between the two methods is exactly these 7 resources; the
ConfigMap and cilium-agent DaemonSet are unchanged, and the agent's
hubble-tls volume is already optional:true so it starts before certgen
runs. Both files and the inline block scalar validated as YAML.
This is the code half of the fix. Rotating the already-leaked live CA +
Hubble certs and scrubbing them from git history is an operator task
tracked on the issue.
Refs #188
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #188 (code half).
What
kubernetes/talos/patches/cilium-cni.yaml(the live Talos bootstrap inline manifest) and its standalone copykubernetes/talos/cilium.yamlembeddedSecret/cilium-ca(RSAca.key) andSecret/hubble-server-certs(RSAtls.key) as plaintext base64 — rendered Helm output fromhubble.tls.auto.method=helm.This PR switches to
hubble.tls.auto.method=cronJob: the two key-bearing Secrets are replaced by the certgen ServiceAccount / Role / RoleBinding / Job / CronJob. The Job generates (and the CronJob renews)Secret/cilium-caand issuesSecret/hubble-server-certsin-cluster at install. No private key material is committed.Derivation / verification
helm template cilium/cilium --version 1.19.2rendered both ways (method=helm vs method=cronJob); the only delta is exactly these 7 resources (2 secrets removed, 5 certgen added). ConfigMap and cilium-agent DaemonSet are byte-identical between methods, so the change is applied surgically and the rest of each manifest is untouched.Not in this PR (operator tasks, tracked on #188)
The keys already leaked into git history, so this PR alone is not sufficient:
Exact procedure + what is needed from the owner is posted on issue #188.
Note: standalone cilium.yaml has drifted from cilium-cni.yaml (ipv6/BGP differ) — flagged on the issue for separate reconciliation.