Compare commits

..
Author SHA1 Message Date
claude-bot 2e535faa56 refactor(cilium): one seed, one value source, no untracked settings
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Cleans up the duplication behind the version drift rather than only the drift.

talos/cilium.yaml is deleted. It was a second, un-indented copy of the same
~2000 lines already embedded in the patch, and nothing read it — the only
references anywhere were two comments in the HelmRelease. It existed to be
readable next to the patch and cost a whole file that could go stale on its
own, which is precisely what happened.

The seed's header now records how to reproduce it: the exact yq and helm
template invocation, and why the two overrides exist (the ServiceMonitor CRD is
absent at bootstrap; the chart's default cert method would render private keys
into git, #188). None of that was written down, which is why reconstructing the
original value set today took a diff-until-the-object-kinds-match exercise
rather than reading a line.

Three settings move from nowhere into the HelmRelease: bgpControlPlane.enabled
and the two native-routing CIDRs. They were live but tracked in no file at all,
applied by hand and held by the kubectl-patch field manager on cilium-config.
Server-side apply left them alone so nothing ever broke — but a cold rebuild
installs only the seed, and would have brought the cluster up with the BGP
control plane off and no session to the UDM, while BGP looked configured
because the peering CRDs are in git.

That is not a free change of ownership. Enabling bgpControlPlane through the
chart emits five companion keys the hand-patch never set, so Cilium has been
running on its own internal defaults for router-id allocation and friends. The
comment says which ones and what to watch.

The per-key "THREE COPIES of this key exist" warnings are gone, replaced by one
note at the top of the values block: these values feed the seed, and nothing
enforces agreement. Two copies now, not three, and the surviving one says how
it is made.

Deliberately not done: a generator script and a CI check to diff the committed
seed against a fresh render. That answers duplication with more machinery and
still leaves a 2000-line generated artifact in the tree. If this should stop
being hand-maintained the real question is whether Talos should carry the CNI
at all, which is a rebuild-path decision, not a scripting one.

bootstrap.sh is untouched. Its talos phase remains dead.
2026-08-28 15:21:19 +02:00
claude-bot 04b94f5313 docs(talos): the cilium patch does not serve bootstrap
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Its header claimed the delete document is a no-op against an empty initial
config, so "same file therefore serves both paths". #581 disproved exactly
that: the delete fails against a base with no inlineManifests, which is why
`bootstrap.sh talos` cannot generate a config and why RUNBOOK-cold-rebuild.md
tells a rebuild to extract only the second, additive document by hand.

The issue was closed and both runbooks were updated. The claim it disproved was
left in the file, where anyone reading the patch is still told it works for
bootstrap.
2026-08-28 15:11:58 +02:00
claude-bot 956734c774 fix(talos): regenerate the Cilium bootstrap seed at 1.20.1
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The inline manifest that seeds the CNI at bootstrap still carried Cilium
v1.19.2 while the Helm release has been on 1.20.x since 2026-08-16. That was my
omission: 8f45e290 moved the chart constraint to >=1.20.0 <1.21.0 and never
carried the version into the seed, on the same day 621e4e25 was editing this
very file for policyMapMax.

It matters because talosctl upgrade-k8s reconciles bootstrap manifests. Its dry
run against the live cluster shows what a Kubernetes upgrade would have done
before this change:

    image: quay.io/cilium/operator-generic:v1.20.1 -> v1.19.2
    bpf-lb-external-clusterip: "true"  -> "false"
    bpf-lb-sock: "true"                -> "false"
    k8s-require-ipv6-pod-cidr: "false" -> "true"

A CNI downgrade across a minor, socket load balancing switched off, and
external-ClusterIP handling switched off, on a running cluster.

The seed is regenerated by rendering the chart at 1.20.1 with the same values
the HelmRelease uses, minus prometheus.serviceMonitor — its CRD does not exist
at bootstrap, which is why the seed never carried one. Everything else now
matches, so the reconcile is materially a no-op: verified key by key against
the live cilium-config, including the three flags above.

The rendered operator digest is identical to the one the live cluster runs,
which is the strongest available check that the values were reconstructed
correctly rather than merely plausibly.

Not changed, and worth recording because it looks like drift and is not: the
live ConfigMap has three field managers (helm-controller, talos, kubectl-patch)
and carries enable-bgp-control-plane, the native-routing CIDRs and the
identity-label filter. The dry run shows all of them as unchanged context —
server-side apply leaves fields owned by another manager alone. They are not at
risk here.

The coupling this exposes has no guard: nothing fails when the seed and the
HelmRelease drift apart, and the gap is only visible in an upgrade-k8s dry run
that nobody runs routinely. A CI check comparing the two would have caught this
in August.
2026-08-28 15:01:43 +02:00
claude-bot 1f3afcd7e3 feat(woodpecker): liveness probe on the agent healthcheck
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The agent was the last workload in this repo running a real service with no
probe of any kind. WOODPECKER_HEALTHCHECK defaults to true and already serves
:3000, so nothing had to be enabled — the port was simply never declared and
never probed. Confirmed against the running pod: /healthz answers 200, / and
/metrics do not.

What it actually asserts is narrower than the name suggests, and narrower than
I assumed when proposing this. State.Healthy in agent/state.go returns false
only when a running workflow has passed its own timeout plus a one-hour buffer;
an agent with nothing to do always answers 200.

So this catches one specific failure — a wedged build holding a slot, which
today nobody would notice — and explicitly does not catch an agent that has
lost its connection to the server. That agent reports healthy while quietly
taking no work, which was the failure I originally wanted to cover. The comment
says so, because a green probe here is easy to misread as "CI is fine".

Liveness only. The agent fronts no Service, so a readiness probe would gate
nothing. Thresholds are short because the upstream condition already carries an
hour of debounce.

Everything else still unprobed is either operator- or chart-owned (Rook, the
authentik outposts, the kube-prometheus-stack sidecars, Multus, cert-manager,
mariadb-operator, velero node-agent) or a sidecar loop with no listening socket
where an exec probe would assert little beyond what container exit already
does.
2026-08-28 14:31:45 +02:00
claude-bot 469cdbc2ee feat: give the remaining unprobed workloads a health check
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Continues the paperless change. Under the split where gatus watches from
outside and Kubernetes watches the pods, a container with no probe at all is
the hole: it stays Ready while dead, and nothing downstream can tell.

Not every one of them got a probe, and the ones that did not are the more
interesting half.

Added:

  whisper, openwakeword       tcpSocket on the wyoming port. These speak a raw
  speech-to-phrase            TCP protocol with no HTTP surface and no health
                              verb, so accepting a connection is all that can
                              be asserted from outside the process. Weak, but
                              the previous state was nothing at all, and a dead
                              server kept being routed speech.

                              speech-to-phrase gets a startupProbe rather than
                              a long liveness delay: it runs --retrain-on-start
                              and retraining takes as long as it takes. The
                              gate allows 10 minutes and only then hands over,
                              so a slow retrain is not mistaken for a hang
                              while a real hang is still caught in a minute.

  ftp-scanner                 tcpSocket on the FTP control port. This is the
                              failure that hides best: if vsftpd dies, the
                              scanner cannot deposit anything and paperless
                              just sees an empty consume directory, which looks
                              exactly like nobody having scanned.

  code-server                 httpGet /healthz. Answers 200 whenever the server
                              is up; its body reports session heartbeat, which
                              is not health — an idle editor reports "expired"
                              and is fine.

  external-secrets            livenessProbe/readinessProbe enabled through the
                              chart's own values, which is also what starts its
                              health server: without them the controller
                              exposes nothing but a metrics port. A wedged
                              reconciler stayed Ready while every ExternalSecret
                              silently stopped refreshing.

Deliberately left alone:

  protonmail-bridge           Already carries a rationale: the bridge can be
                              offline while the user re-authenticates, and
                              paperless retries mail fetches itself. A liveness
                              probe would restart it mid-re-auth. That is a
                              considered decision, not an omission.

  bridge-tls-proxy            socat binds 127.0.0.1:11143. Probes are dialled
                              by the kubelet against the pod IP, so there is
                              nothing for it to reach.

  mariadb-operator            Its chart exposes no probe values and no health
                              bind address. Forcing them through a postRenderer
                              would break on the next chart bump, so it stays
                              unprobed and this records why.

  ak-outpost-* (11)           Pod specs are owned by the goauthentik.io
                              operator; nothing here can add to them.

Validated with Flux's field manager against the live cluster.
2026-08-28 13:24:05 +02:00
10 changed files with 613 additions and 2247 deletions
@@ -210,6 +210,28 @@ spec:
- name: vscode
containerPort: 8080
protocol: TCP
# /healthz answers 200 whenever the HTTP server is up; its body
# reports session heartbeat state, which is not health — an idle
# editor reports "expired" and is perfectly fine. So this asserts
# that code-server is serving, not that anyone is using it.
#
# Liveness matters here more than elsewhere in this pod: code-server
# shares the Home Assistant config volume, and a wedged editor is
# otherwise invisible until someone tries to open it.
livenessProbe:
httpGet:
path: /healthz
port: vscode
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /healthz
port: vscode
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
volumeMounts:
- name: config
mountPath: /config
@@ -24,6 +24,22 @@ spec:
- name: wyoming
containerPort: 10400
protocol: TCP
# Wyoming is raw TCP with no HTTP surface and no health verb, so
# accepting a connection is all that can be asserted from outside the
# process. Weaker than an HTTP probe, but the previous state was none
# at all: a dead server stayed Ready and kept being routed to.
livenessProbe:
tcpSocket:
port: wyoming
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
tcpSocket:
port: wyoming
initialDelaySeconds: 15
periodSeconds: 10
timeoutSeconds: 5
resources:
requests:
cpu: 100m
@@ -70,6 +70,27 @@ spec:
- name: consume
mountPath: /home/scanner
subPath: scanner
# FTP has no health verb, so the control port accepting a connection
# is what can be asserted. It is enough for the failure that matters:
# if vsftpd dies the scanner cannot deposit anything and nothing else
# in the chain notices, because paperless only ever sees an empty
# consume directory — which is indistinguishable from nobody having
# scanned.
#
# Only the control port is probed. The passive range is negotiated
# per transfer and is not listening at rest.
livenessProbe:
tcpSocket:
port: ftp-ctrl
initialDelaySeconds: 15
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
tcpSocket:
port: ftp-ctrl
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
resources:
requests:
cpu: 20m
@@ -34,6 +34,32 @@ spec:
- name: wyoming
containerPort: 10300
protocol: TCP
# Wyoming is raw TCP with no HTTP surface and no health verb, so
# accepting a connection is all that can be asserted from outside the
# process — weaker than an HTTP probe, but the previous state was
# none at all.
#
# A startupProbe rather than a long liveness delay: this container
# runs with --retrain-on-start, and retraining against the Home
# Assistant entity list takes as long as it takes. The startup gate
# allows up to 10 minutes (60 x 10s) and only then hands over to
# liveness, so a slow retrain cannot be mistaken for a hang, and a
# genuine hang after startup is still caught within a minute.
startupProbe:
tcpSocket:
port: wyoming
periodSeconds: 10
failureThreshold: 60
livenessProbe:
tcpSocket:
port: wyoming
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
tcpSocket:
port: wyoming
periodSeconds: 10
timeoutSeconds: 5
volumeMounts:
- name: models
mountPath: /models
+21
View File
@@ -29,6 +29,27 @@ spec:
- name: wyoming
containerPort: 10300
protocol: TCP
# Wyoming is a raw TCP protocol with no HTTP surface and no health
# verb, so accepting a connection is the only thing that can be
# asserted from outside the process. That is weaker than an HTTP
# probe but strictly better than the previous state of none at all,
# where a dead server stayed Ready and Home Assistant kept routing
# speech to it.
#
# The liveness delay is generous because the model is loaded at
# startup from the data volume.
livenessProbe:
tcpSocket:
port: wyoming
initialDelaySeconds: 90
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
tcpSocket:
port: wyoming
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
volumeMounts:
- name: data
mountPath: /data
+25
View File
@@ -94,6 +94,31 @@ spec:
envFrom:
- secretRef:
name: woodpecker-secret
ports:
# WOODPECKER_HEALTHCHECK defaults to true and serves :3000, so this
# port already existed; declaring it just lets the probe name it.
- name: health
containerPort: 3000
protocol: TCP
# /healthz answers 500 ONLY when a running workflow has passed its own
# timeout plus a one-hour buffer (agent/state.go, State.Healthy). An
# idle agent always answers 200.
#
# So this catches a wedged build holding a slot, which is otherwise
# invisible — and it does NOT catch an agent that has lost its server
# connection, which reports healthy while quietly taking no work. Do
# not read a green probe here as "CI is fine".
#
# Liveness only: the agent fronts no Service, so readiness would gate
# nothing. The upstream condition is already debounced by that hour,
# which is why the thresholds here are short.
livenessProbe:
httpGet:
path: /healthz
port: health
initialDelaySeconds: 20
periodSeconds: 30
timeoutSeconds: 5
resources:
requests:
cpu: 25m
@@ -33,6 +33,13 @@ spec:
remediation:
retries: 3
values:
# THESE VALUES ALSO FEED THE BOOTSTRAP SEED. talos/patches/cilium-cni.yaml
# is rendered from this block; the command is in that file's header. Change
# something here and the seed is stale until it is re-rendered, and nothing
# will tell you — the only place the drift shows is a `talosctl upgrade-k8s
# --dry-run`, which is how a 1.19.2 seed survived a month behind a 1.20
# chart.
#
# Rolls the agent DaemonSet when cilium-config changes. Cilium reads
# datapath options once at agent start and the chart puts no config
# checksum on the pod template, so without this a changed ConfigMap leaves
@@ -50,8 +57,6 @@ spec:
# and every identity costs one entry per allowed (direction, proto, port)
# in every endpoint's policy map, cluster-wide. Nothing selects on these
# labels in policy, so they buy identity churn and nothing else.
# Keep in lockstep with kubernetes/talos/patches/cilium-cni.yaml and
# kubernetes/talos/cilium.yaml.
labels: '!woodpecker-ci\.org/.* !step$'
ipv6:
enabled: true
@@ -76,10 +81,6 @@ spec:
# cilium_bpf_map_pressure. Sized at map creation, so it takes an agent
# restart.
#
# THREE COPIES of this key exist — here, talos/patches/cilium-cni.yaml
# and talos/cilium.yaml. The latter two are what a cold rebuild installs
# from. Nothing enforces agreement; grep bpf-policy-map-max before
# changing it.
policyMapMax: 65536
# VLAN ids the eBPF datapath is allowed to pass. Without this it drops
# VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
@@ -140,3 +141,26 @@ spec:
enabled: true
externalIPs:
enabled: true
# These three were live but tracked nowhere: applied by hand, owned by the
# `kubectl-patch` field manager on cilium-config, absent from this file and
# from the bootstrap seed. Server-side apply left them alone, so nothing
# broke and nothing noticed — but a cold rebuild installs only the seed, and
# would have come up with the BGP control plane off and no BGP session to
# the UDM at all. Declaring them here puts them under Flux and carries them
# into the seed, which is generated from these values.
#
# The values match what the cluster runs, but this is not purely a change of
# ownership: enabling bgpControlPlane through the chart also emits five
# companion keys the hand-patch never set — bgp-secrets-namespace,
# bgp-router-id-allocation-mode and -ip-pool,
# enable-bgp-control-plane-status-report, enable-bgp-legacy-origin-attribute.
# Cilium has been running on its own internal defaults for those. They
# should agree with the chart's, but router-id allocation is the one worth
# watching on apply: if it disagrees, the sessions re-establish.
#
# The peering itself is CRDs in post-crds/cilium-bgp.yaml — that was always
# in git; only the switch that starts the control plane was not.
bgpControlPlane:
enabled: true
ipv4NativeRoutingCIDR: 10.0.0.0/8
ipv6NativeRoutingCIDR: fd00::/8
@@ -33,3 +33,16 @@ spec:
crds: CreateReplace
remediation:
retries: 3
values:
# The chart ships these disabled, so the controller ran with no probe of
# any kind: a wedged reconciler stayed Ready and every ExternalSecret would
# simply stop refreshing, which surfaces only when a consumer needs a
# credential that has since rotated.
#
# Enabling them also starts the health server — the chart derives
# --health-probe-bind-address from these values, and without them the
# controller exposes nothing but its metrics port.
livenessProbe:
enabled: true
readinessProbe:
enabled: true
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long