#797 added build-vmtest with gcr.io/kaniko-project/executor:v1.23.2-debug,
the archived Google image the other nine steps left in August (60f231f8).
It passed on master only because that image was still cached on the node;
the branch push of the controller PR scheduled it elsewhere and the step
died before its first command. Same fork pin as the rest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
netbird-proxy-services reconciles the NetBird reverse-proxy services that
nobody managed so far: one TLS-passthrough service on :443 per host of every
Ingress carrying dns.renner.uno/public, targeting Traefik ClusterIP through
a subnet resource over the service CIDR. SNI and Host reach Traefik
untouched, the proxy terminates nothing, no domain resource is involved --
so the dns-interceptor pinning bug that wedges the proxies on every
routing-peer loss has nothing to pin on.
Ownership is derived from NetBird, not from local state: managed iff mode is
tls and the target is the controller subnet resource. The ten hand-made
HTTP/domain services are reported as held and never touched; migration is
deleting one of them, the controller creates the passthrough on its next
pass. The PAT is read from OpenBao per pass through a read-only role
(policies/netbird-proxy-services.hcl, applied once by hand like the rotator
roles). Standard library only; unprivileged image built alongside heating-can.
Validated against the live management (ONCE=1 through the rotator role):
network, resource, router and policy created, 10 hosts held, one host that
had no service at all (llm.renner.uno) created and answering behind its
Authentik gate; second pass 0 created / 0 updated / 0 deleted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every loss of the routing peer a proxy has pinned wedges its route
manager; the flood watchdog restarts the container within 2 s and the
public edge is back after one failed probe round. Inside the 600 s
cooldown the restart is suppressed and the URL watchdog three one-minute
rounds are the only recovery: 3.5 min of 000 on every public domain,
measured 2026-09-07 10:28 on proxy-2 ("restart suppressed, last restart
567s ago"). The morning node roll hit the same suppression on both
proxies at 06:58 and survived only because the routes recovered by luck.
Rolling reboots lose one peer every few minutes by design, so with 600 s
the second roll is always the unlucky one. A minute keeps the loop
protection (one restart per probe interval at most).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Phase 1 (#802) put the ServersTransport under post-crds and
infrastructure-crds has applied it. The copy in crowdsec/lapi-ingress.yaml
goes; the Ingress stays (core kind). With it, the infrastructure apply set
holds no traefik.io consumer any more, which is what a cold rebuild needs.
Prune: infrastructure reconciles first and skips the object if the last
applier label names infrastructure-crds; otherwise it deletes and
infrastructure-crds recreates it seconds later. Only the public LAPI
ingress used by the external bouncers is behind that transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Since the host-device rollout (#801) Rook cannot finish a CephCluster
reconcile: it runs a canary pod with OSD placement to read the cluster
CIDR off the attachment, the OSD on that node already holds the NIC, and
the canary fails "error adding container to network ceph-san" until the
15-minute discovery timeout, then the next reconcile repeats it. Under
macvlan the canary could sit beside the OSD; a whole-device attachment has
exactly one consumer per node.
Discovery is skipped when addressRanges.cluster is set (Rook: "if user
specifies CIDRs in the network config, that is the definitive source of
truth"). The value is what discovery produced and what ceph config already
holds, so no daemon changes. The old comment claiming addressRanges is
honoured only under provider: host was wrong — the CRD documents it for
host and multus.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every node has a second, address-less NIC (enp87s0) up on the untagged
fallback network, which shares the mgmt VLAN's broadcast domain. With
the Linux default arp_ignore=0 the host answers ARP for the node IP from
that NIC too, so the UDM can learn 10.2.40.x against enp87s0's MAC.
Inbound IPv4 then arrives on a device Cilium is not attached to, never
reaches cil_from_netdev, and is routed straight back to the UDM by the
default route — a loop that made every IPv4 LoadBalancer VIP on the
affected node unreachable for 13.5 hours. IPv6 was untouched because
NDP only answers on the interface that owns the address.
The flux was measurable on all three nodes: an ARP request for a node
IP from a neighbour returned replies from two MACs. Which node's entry
had actually flipped at the UDM was a race, and it moved during the
diagnosis.
arp_ignore=1 answers only when the target address is configured on the
receiving interface; arp_announce=2 stops the host from sourcing ARP
from an interface that does not own the address. Applied live with
talosctl on all three nodes before this commit and verified with
enp87s0 up: each node now answers from its mgmt0 MAC only.
A node property, so it lives next to the existing enp87s0 sysctls in
each machine patch rather than in the site patch.
Refs #793, #576
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
All three OSD prepare jobs on the Hetzner rehearsal finished with zero
OSDs:
cephosd: skipping device "sdb1": ["Used by ceph-disk"]
The partition is exactly what the machine config asked for (PARTLABEL
r-ceph-osd, plain Linux partition type). ceph-volume rejects it anyway —
its ceph-disk-member test is `"ceph" in PARTLABEL or PARTTYPE in
ceph_disk_guids` (ceph_volume/util/device.py) — so any label containing
"ceph" reads as a legacy ceph-disk partition and is never provisioned.
Talos derives the label from the RawVolumeConfig name (r-<name>), so the
name changes: ceph-osd -> osd-data, label r-osd-data. This is why the
rebuild branch called it osd-data; the #798 review renamed it back for
symmetry with the node label without this failure being known.
Production is unaffected today: the MS-01 OSDs match on the by-id half of
devicePathFilter and no r-ceph-osd partition exists there. The partlabel
half is the migration target and would have failed on the first
repartitioned drive.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A cold rebuild stops permanently at the infrastructure Kustomization:
ServersTransport/crowdsec/crowdsec-lapi dry-run failed: no matches for
kind "ServersTransport" in version "traefik.io/v1alpha1"
Flux dry-runs the whole apply set, so this one unknown kind rejects all of
it — including the Traefik HelmRelease that installs the CRD. Measured on
the Hetzner rehearsal from master: zero HelmReleases, zero CRDs. The
public LAPI endpoint (2026-09-02) landed after the post-crds sweep and put
a traefik.io consumer back into the layer that creates traefik.io.
Phase 1 only adds the object under post-crds; crowdsec/lapi-ingress.yaml
keeps its copy until infrastructure-crds has applied once, because both
Kustomizations prune and a one-commit move deletes before the new owner
applies. Phase 2 removes the original.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The NetworkAttachmentDefinition switches from a macvlan on a kernel
interface name to host-device on the role name `ceph-san`. The NIC is
dedicated to Ceph and has exactly one consumer per node, so it is moved
into the OSD pod whole — real MAC, real MTU — and returns to the host when
the pod goes away. No hardware name appears in this file any more; every
machine renames its own link at boot (previous commit).
ROLLOUT — not atomic with the machine config. Between this flip and a
node's reboot, an OSD restart on that node fails: the NAD asks for
ceph-san and the running node only has the old name. Merge only inside a
maintenance window, then drain + noout + reboot k8s-1/2/3 back to back.
Running OSD pods keep their existing attachment until restarted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only the Talos machine config knows hardware. Each machine renames its own
Ceph cluster-network link to `ceph-san` at boot, so shared manifests can
address the role and no kernel interface name leaves this directory:
- k8s-1/2/3 (MS-01): a udev rule matches the X710 port 1 by burned-in MAC
and renames it before the network stack opens links. The LinkConfig then
names ceph-san (up, mtu 9000). A rule applied to a RUNNING node cannot
rename (Resource busy) and takes effect at the next reboot, so this
change is inert on production until each node is rebooted — that is the
planned rollout, node by node.
- vm-1..4 (qemu site): one NIC only, so the segment is VLAN 30 named
ceph-san directly. The DHCPv4Config on the parent is load-bearing:
declaring any network document stops the implicit DHCP that addresses
the node.
- hcloud-1/2/3 already carry the same shape (udev by PCI bus path) since
the rebuild-tooling merge.
Validated on Hetzner with the host-device attachment: OSD pods held the
real fabric MACs, Ceph reached HEALTH_OK.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 19.x pin existed to keep this cutover deliberate and tied to the
Rook operator version. That coupling has already come apart in the
dangerous direction: Rook floating constraint pulled v1.20 in June, and
with it cephcsi v3.17.1, whose client libraries are Ceph 20.2.1 — so
every CSI pod has been talking to 19.2.6 mons with a client a whole
major ahead of them.
That is not academic. The 19.2.6 mons mint auth keys as aes256k, and
the 20.2.1 client cannot decode them ("auth: failed to decode key").
Nothing has broken only because the four CSI users predate the cipher
change and still carry legacy aes keys — which Ceph itself flags daily
as HEALTH_WARN (AUTH_INSECURE_CLIENT_KEY_TYPE, 5 entities). The moment
one of those is recreated — ceph auth rotate, a deleted Secret, Rook
resyncing — the provisioner stops and no PVC binds. A cold rebuild hits
it immediately: a fresh cluster has no legacy keys at all.
So the cutover happens now, with the version the client already is.
Rook v1.20 supports Tentacle from v20.2.1; v20.2.0 is excluded
deliberately — upstream reports data corruption there with read
affinity enabled. The two imagepolicy-marked lines are bumped in the
same commit so image automation finds nothing left to do.
Rollout is Rook-managed and rolling: osdMaxUpdatesInParallel: 1 and
upgradeOSDRequiresHealthyPGs: true bound it to one OSD at a time with
clean PGs between steps. Expect mons, then mgrs, then three sequential
OSD restarts with recovery between them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
None of this is reconciled by Flux — bootstrap.sh, the vmtest Job and
the Talos hcloud patches are operator tooling, applied only when a
rebuild test runs. Zero effect on the live cluster from merging.
New: an hcloudtest phase that provisions three cpx42 servers with
private cluster/storage networks, a this-host-only firewall and one raw
OSD volume per node, then hands over to the ordinary talos phase through
the same four variables vmtest uses. Servers carrying the test label are
adopted and rebuilt from the Talos snapshot instead of recreated —
capacity, not cleanliness, is the scarce resource. Network ranges and
volume sizes are derived from what the cluster itself declares (the site
patch and the machine patches), never stated a second time.
hcloudclean deletes everything by label.
The vmtest Job clones from the GitHub mirror instead of in-cluster
Gitea: a rebuild rehearsal cannot depend on the thing whose loss it
rehearses, and a run died on resolving gitea.renner.uno before the
first VM booted.
bootstrap.sh hardening from a week of rehearsals: the OpenBao unseal
loops until the StatefulSet converges instead of unsealing whichever
pods existed at one instant, require_secrets takes a subset, secrets can
be streamed into a waiting run, retries around talosctl bootstrap and
server placement, and node waits print progress and refuse an expected
count of zero.
import-cluster-access.sh keeps its default AWS profile but stops
overriding one the caller already chose.
Known boundary, documented in site-hcloud.yaml: from master the hcloud
site validates provisioning through cluster-Ready; the SAN path needs
the follow-up rollout PR that renames the storage NIC by role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One unknown kind fails a server-side dry-run for the WHOLE apply set,
including the very HelmRelease that would have installed the missing
CRDs. On a running cluster every CRD exists, so this never showed; a
cold rebuild deadlocks on it immediately.
Three groups move, content unchanged:
ExternalSecrets and Traefik Middlewares out of infrastructure into
post-crds, which reconciles after the operators. The flux-git
ExternalSecret also changes creationPolicy Merge -> Orphan with
deletionPolicy Retain: Merge never CREATES the target secret, so on a
fresh cluster the Gitea credential handover silently did not happen.
ServiceMonitors/PodMonitors out of their charts and app sets into
apps/monitoring-rules, which applies after kube-prometheus-stack owns
the CRDs. Rook additionally stops creating its own two monitors
(spec.monitoring.enabled: false) — creating them is fatal to its
node-daemon reconcile when the CRD is missing, which held the
CephCluster in Progressing and with it the whole apps layer. Both
objects are reproduced field for field off the live cluster; the
ceph-exporter DaemonSet and the metrics endpoints are untouched.
Cilium is the worst case: its chart aborts rendering outright on the
missing CRD, and cilium is one of the seven infrastructure
healthChecks — the lowest layer in the cluster waiting on the highest.
Steady-state effect: object ownership moves between Kustomizations;
specs are identical. Verified on local vmtest runs and the Hetzner
throwaway cluster.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gitea runs inside the cluster, so during a cold rebuild every image that
only exists at gitea.renner.uno is unpullable — a rehearsal died with the
cni-plugins installer in ImagePullBackOff, no macvlan binary, no ceph-san
attachment, CephCluster stuck at "Configuring Ceph Mons", OpenBao waiting
forever on a PVC. The failure surfaced four layers from its cause.
The CI images move to konairius/, published by the build pipeline to
Gitea AND ghcr.io with one digest. The manifests keep pointing at Gitea;
the node-level fallback (machine.registries.mirrors, written by
bootstrap.sh at install time) tries Gitea first and ghcr second, and
containerd keeps the repository path when it falls through — so the same
path resolves on both. Steady state never leaves the network.
The Multus, whereabouts and cni-plugins-installer DaemonSets also gain
priorityClassName system-node-critical. Upstream ships them at priority
0: on a rebuild two nodes sat at 99% memory requests when the DaemonSet
rolled, both multus pods stayed Pending, and every pod needing a
secondary attachment waited on an annotation that could never come — with
nothing reporting why.
Rollout effect on the live cluster: the three CNI DaemonSets restart once
(existing pods keep their networking; only new attachments wait), and the
three init containers pull the new tag on their next pod restart. The
ghcr side is proven by the Hetzner throwaway runs, which pull through the
fallback; eyeball gitea.renner.uno/konairius packages before merging.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by rebuild rehearsals; each stopped a cold start outright and none
changes behaviour on a running cluster.
infrastructure-ceph-cluster no longer dependsOn infrastructure. That
edge closed a cycle no new cluster could escape: infrastructure is Ready
only once its openbao healthCheck passes, openbao-0 runs only once its
PVC binds, and the PVC wants the ceph-block StorageClass defined in this
very Kustomization. infrastructure runs wait: false, so everything this
path needs is applied long before readiness; on an empty API server it
self-heals on retryInterval like post-crds already does.
home-assistant gains dependsOn infrastructure-cnpg-barman: its
ObjectStore kind arrives with the Barman plugin, and one unknown kind
fails the server-side dry-run for the whole apply set — taking
speech-to-phrase, zone-configurator and matter-server down with it.
Every other ObjectStore user already declares the edge.
woodpecker-ci namespace moves to infrastructure/namespaces: its
CiliumNetworkPolicies ship with infrastructure-crds, which runs before
every app KS, so declaring the namespace in the workload KS inverted the
order and held infrastructure-crds — and the twenty app Kustomizations
behind it — on "namespaces woodpecker-ci not found". Same reasoning as
its sibling namespace, previously not carried across.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The vmtest Job passes VM_STATE=/work/vm to aim the qcow2 images at the
150Gi emptyDir it sizes for them. bootstrap.sh overwrote it
unconditionally, so the images landed on the container's writable layer
instead and the sizeLimit guard never applied to anything.
Default to the same path as before when the caller sets nothing, so a
plain `bootstrap.sh vmtest` is unchanged.
Two guards come with it, because the value is now caller-controlled:
* a length cap, since the provisioner puts a qemu monitor socket under
this directory and sun_path stops at 108 bytes. Too long used to be a
silent failure to boot; it is now an error. Both the old default (21
bytes) and the Job's /work/vm (8) are well inside it.
* a shape check, because cleanup_vmtest runs `sudo rm -rf "$VM_STATE"`.
That was safe by construction while the path was hardcoded and is not
once a caller picks it.
This does not by itself reduce drive wear — an emptyDir also lives on
/var, so both paths are on the same device. It restores the size guard
and makes it possible to point VM state at a different device later.
VMTEST_FULL and VMTEST_KEEP, which the Job also sets, are still not read
anywhere in this script. Left alone rather than guessed at: the Job is
applied out of band and is not in this repo, so the intended semantics
are not visible from here.
Refs #754
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The evaluation is over. Removing clusters/homelab/paperless-eval.yaml is
the designed teardown: that Kustomization is prune: true with no
prune-disabled annotation anywhere underneath, and its own comment says
being able to delete the namespace by removing the file is the point.
Removed alongside it, so nothing recreates the namespace or points at it:
kubernetes/apps/paperless-eval/ the manifests
clusters/homelab/paperless-eval.yaml the Flux Kustomization
network-policies/paperless-eval-ns.yaml and its kustomization entry
namespaces/namespaces.yaml the Namespace object
network-policies/ai-ns.yaml the llama-embed ingress grant
post-crds/openbao-clustersecretstore.yaml the ESO namespace grant
Production backups are unaffected. The eval Cluster read
s3://renner-uno-k8s-cnpg/pprl-pg as a recovery source with
isWALArchiver: false and no retentionPolicy, so it never wrote to the
production prefix and deleting it removes nothing from the store.
The model weights on the namespace's PVCs were already deleted by hand
before this change; the volumes go with the namespace.
scripts/paperless-eval/ is deliberately left in place — it is the tooling
that drove the evaluation, not part of the namespace, and it is only
useful if the namespace is ever recreated from these manifests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cilium-health reports 0/3 reachable on every node, with Node 1/2 and
Endpoints 2/2 — exactly one of the two node probes fails, including
against localhost.
The Node objects carry only InternalIP=10.2.40.x, because kubelet's
nodeIP.validSubnets lists only the IPv4 mgmt subnet. With no IPv6
InternalIP to take from the Node object, Cilium falls back to detecting
the node's IPv6 from mgmt0 and picks the SLAAC address derived from the
dynamic provider prefix. It records that in CiliumNode and probes it
until the agent restarts, so a prefix change leaves every agent probing
an address that no longer exists:
cilium-health: 2003:d6:9718:3b02:5a47:caff:fe74:7027
mgmt0 now: 2003:d6:9713:b202:5a47:caff:fe74:7027
The interface identifier is the same EUI-64; only the prefix moved. This
is not a one-off — the prefix was 2003:d6:9744:4802 in May and
2003:d6:9718:3b02 in August.
The GUA was never meant to be a control-plane address; that is what the
ULA is for, and BGP already peers on fd40::/64. Listing it in
validSubnets gives the Node a stable IPv6 InternalIP, which Cilium then
uses in preference to its own detection — the same path that already
makes CiliumNode's IPv4 InternalIP agree with the Node object.
Verified fd40::/64 SLAAC addresses are present on mgmt0 on all three
nodes, so no node is left without an address in a listed subnet.
The data path is unaffected either way: Endpoints probes are 2/2 and
conntrack is clean. What this fixes is the health signal, which is
permanently red today and therefore useless for spotting a real outage.
Refs #789
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both socat containers exit 1 on the first datagram they cannot deliver.
While no ready pod backs home-assistant-mdns, its ClusterIP has no
backend and the sendto fails with EHOSTUNREACH, so every Home Assistant
restart costs both containers a kubelet restart — 52 in 10 days.
Run each socat under a shell loop instead. A target that is briefly gone
now costs a two-second gap in what is already a best-effort cache-refresh
path, rather than a restart. Re-running from scratch also re-resolves the
ClusterIP name and rejoins the multicast group.
socat has no option that avoids the wrapper: -s ("sloppy") downgrades the
exit status to 0 but still ends the transfer loop, so the kubelet restarts
the container anyway and the failure stops being visible in the exit code.
The address-level retry/forever options govern opening an address, not
writing to one. Both verified against the pinned image.
The loop traps TERM and forwards it to the backgrounded socat, so the pod
still terminates in ~0.1s; with strategy: Recreate a stalled terminate
would delay every rollout by the full grace period.
Trade-off: a permanently broken relay no longer surfaces as a restart
count. The restart line on stderr is the replacement signal.
Refs #787
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both carried a hand-written exemption in image-automation/README.md. The
exclusion list is for images whose tag format makes a policy impossible;
these two publish clean semver streams and were enrollable all along.
Excluding them on blast-radius judgement left them unowned, and they
drifted.
terminus 0.68.0 -> 0.72.0 (both containers)
trmnl-renderer 0.9.5 -> 0.10.3
That drift is not cosmetic. 0.69-0.72 include "Fixed device provisioner
answer blank API key when device exists", "Fixed Display API image name
to use timestamp instead of checksum" and "Fixed API log create action to
authenticate with API key" — the provisioning, redraw and log paths that
the entrance panel's onboarding was debugged and worked around by hand on
2026-09-05, on a server a month behind the fixes.
The renderer's second stated reason, no upstream multi-arch manifest, is
not an obstruction: the reference already names the architecture, so the
policy pins nothing the tag did not.
Adds apps/trmnl/README.md: which firmware artefact to use for a manual
flash (the mirror carries the C3 build, this panel is S3), what to check
after an image bump, and the device endpoint's scheme constraint.
Watch the panel after this lands: a bad renderer or terminus bump shows
up as a blank or corrupted screen, not as a failed rollout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cleartext LoadBalancer was justified with "the firmware cannot be
handed the internal CA". There is no internal CA in that path:
letsencrypt-cluster solves DNS-01 against the renner.uno Cloudflare zone,
so a LAN-only name gets a publicly-trusted certificate without ever being
publicly reachable — which is why trmnl.renner.uno has always served an
issuer CN=YR2 cert.
trmnl-device.renner.uno therefore moves onto Traefik with its own
Ingress, and API_URI follows it to https so the image_url handed back on
/api/display is fetched over the same scheme the device polls with. The
name moves rather than being duplicated: external-dns runs --policy=sync
and would flap the record between the Traefik VIP and the LB address if
both an Ingress and the Service claimed it.
The LoadBalancer stays, without a hostname. http://10.2.99.89 remains a
path to /api/setup that depends on neither Traefik nor cert renewal, and
the captive portal takes a bare address, so it is the recovery route if
TLS is ever what breaks the panel.
Note Traefik's web entrypoint carries a global http -> websecure
redirect, so http://trmnl-device.renner.uno now answers 308 rather than
serving. The device is provisioned with the https URL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Hermes probe reported "never became healthy" after an hour. It had not
failed to load — it had never been scheduled, because both iGPUs were busy with
other jobs and there are only two.
Same blind spot as the eval pods sitting in Init: from outside, "waiting" and
"will never start" look identical, and the message blamed the model. It now
prints the pod phase every two minutes and exits immediately with the
scheduler's own reason when it cannot place the pod.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The measurements were coupled. One request that corrupts the slot leaves it
returning a single repeated token for everything after it, so a run counted
that fault once per remaining document. "Qwen3-8B failed 4 of 5" was really
"document 1970 succeeded, document 363 broke the slot, and the remaining three
measured the wreckage".
Proved mid-run rather than inferred: with two documents failed, the model
answered "Nenne die Hauptstadt von Frankreich" with
" \" \" \" \" \" \" \" \" \" \" ...
llama-server clears a slot on POST /slots/0?action=erase, but answers 501
without --slot-save-path, so the flag goes on the sidecar and eval.py calls it
after every AI document. It costs the prefix cache and some prefill speed,
which is the right trade for numbers that mean what they say.
Also worth recording: a synthetic reproduction of the poisoning failed. The
real paperless request corrupts the slot; my own 5000-token probe with the same
14-field schema does not, and the canary stayed healthy through it. Whatever
triggers this lives in the parts of a real request I was not reproducing — the
system prompt, the RAG context, the full taxonomy lists. Third time today that
a synthesised test passed where the real one fails, so the harness should stop
trying to stand in for the real request and instead detect when it has gone
wrong, which is what the canary now does.
Re-screening qwen3-8b to find out whether it scores more than one of five once
the documents are actually independent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The canary ran before the documents, every tenth document, and on abort. A
five-document screen whose first document succeeds trips none of those after
the first, so it could finish having never re-verified the instrument it just
used — which is the exact hole the canary exists to close.
It now runs at the end of every AI engine pass regardless of outcome. Healthy
before and broken after means the failures in between are one fault, and the
run says so rather than leaving it to be worked out later.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third false verdict from this script in a row, all the same shape: it asked
whether "Paris" came back, and a working model failed twice for reasons that
had nothing to do with the defect.
read only `content` -> a thinking model writes to reasoning_content
64-token budget -> a thinking model never reaches the answer
Both were my bounds, not the model's, which is the error that has run through
this whole exercise.
So it now tests the defect itself. The signature is a single token repeated —
" de de de", " phosphorus phosphorus", whitespace — and a run is BROKEN when
one token is more than half the output, EMPTY when nothing comes back. That
survives truncation, does not care which field the model writes to, and does
not require the model to finish or to be right.
The general lesson, since it cost three attempts: an implausible reading is the
only kind that gets checked. BROKEN at 154 prompt tokens was obviously wrong,
so it got investigated. A boundary reported at 4000 would have been believed —
it would have fit the expectation. That is exactly how the 768-token cap, the
Q8 comparison and the 35B threshold all survived as long as they did.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First run of probe-length.sh called Qwen3-8B BROKEN at 154 prompt tokens. No
model degenerates at 154 tokens — the verdict was wrong, not the model.
It read only `content`, and a thinking model puts its answer in
`reasoning_content` and leaves content empty. Empty content is not a broken
model, and this is the same mistake that made Qwen3.6 look slow and broken this
morning before --reasoning off went in.
Now reads both fields.
Noting it because the script exists specifically to stop me judging models on
bad readings, and its first reading was bad. A tool that checks the instrument
still has to be checked itself — the 154 is what gave it away, since it was
implausible rather than merely unwelcome.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
claude-botandClaude Fable 5.1
k.renner
security-bot