external-dns v0.22.0 changed the default --annotation-prefix from
external-dns.alpha.kubernetes.io/ to external-dns.kubernetes.io/ with no
fallback, and its release notes lead with the consequence: "This change can
delete all your DNS records."
The chart floated onto it at 2026-09-11 23:47:56 (appVersion 0.21.0 ->
0.22.0). Two hours later, at 2026-09-12 01:47:59, the unifi provider logged
`client deleted record` for every name that depended on the annotation:
ha.renner.uno mqtt.renner.uno ldap.renner.uno
syslog.renner.uno ftp.paperless.renner.uno
syncthing.paperless.renner.uno
A, AAAA and the TXT ownership records, and no `client created record` since.
Every one of those is a LoadBalancer Service, whose hostname external-dns
can only learn from the annotation — so the source went silent and
--policy=sync removed the records. Ingress-sourced names take their host
from spec.rules[].host and were untouched, which is exactly the split
observed: source_endpoints_total is 36 with source_errors_total 0.
These six annotations were the only alpha-prefixed external-dns annotations
in the repo, so moving them to the current prefix is the whole fix. Pinning
the prefix back with --annotation-prefix would keep us on a deprecated
default that upstream intends to drop.
Scanning has been broken since: the Brother resolves ftp.paperless.renner.uno
rather than using the address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LB address was declared twice: on the Service as io.cilium/lb-ipam-ips,
and again as a literal PASV_ADDRESS in the ConfigMap generator.
The copy was not doing anything. vsftpd listens on a single IPv6 socket, so
an IPv4 client arrives as ::ffff:a.b.c.d and vsftpd answers PASV with
0.0.0.0 regardless of the configured address — the only such session in 30
days of logs got `227 Entering Passive Mode (0,0,0,0,82,109)` while
PASV_ADDRESS=10.2.99.82 was set. IPv6 clients negotiate EPSV, which carries
a port and no address at all; that is what the scanner uses.
Removing the literal falls back to the image default of 0.0.0.0, which is
the reply clients already receive, so behaviour is unchanged and the address
now exists in exactly one place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vsftpd 3.0.5 segfaults its standalone listener on roughly 0.3% of the
connections it accepts: the parent closes the accepted socket before it
registers the forked child's pid, so a child that exits inside that window
makes the SIGCHLD path look up an unregistered pid, get NULL, and
dereference it in hash_ip(). PID 1 is vsftpd, so the segfault takes the
container with it.
The tcpSocket probes added in 469cdbc2 opened ~8600 connections a day
against that, which turned a latent crash into a restart every ~25 minutes
and left the scanner hitting CrashLoopBackOff windows. Measured on the live
pod: 52 restarts in 21h, one node-side `vsftpd[...]: segfault at 0` per
restart, starting the hour the probes rolled out.
Assert the listening socket from inside the container instead. The probe
still catches the failure it was added for — vsftpd not serving port 21 —
without being a client of the process it watches.
vsftpd 3.0.5 is upstream's last release (2021) and Alpine ships it
unpatched in 3.21 (3.0.5-r2) and edge (3.0.5-r3), so there is no version
bump that fixes this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TokenStorageMethod is Encrypted and the container has no OS secret store,
so LibationCli falls back to minting a portable master key at
/config-internal/libation-master.key -- a path that is not a mounted
volume. The key is therefore new on every run and discarded at exit,
which is why the entrypoint logs LAST-RESORT on each pass.
The entrypoint looks for the key at ${LIBATION_CONFIG_DIR}/
libation-master.key, i.e. on the PVC, and copies it into place when
found. The key is now stored in OpenBao alongside the other bootstrap
files, so seed it like them: without it a rebuilt PVC would hold tokens
encrypted under a key that no longer exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image entrypoint stages config into /config-internal, which is not a
mounted volume, and it does so by copy:
init_config_file() {
...
cp ${FULLPATH} ${LIBATION_CONFIG_INTERNAL}/
Only LibationContext.db is symlinked to the PVC. AccountsSettings.json is
therefore read from the PVC at startup and written back nowhere, so every
token refresh Libation performs is discarded when the container exits. On
the live PVC the file still carried its 2026-05-14 seed timestamp, four
months after deployment, while the DB beside it was current.
Pass args instead of letting the entrypoint run its own loop. The
entrypoint completes its setup and then execs what it is given, so the
staged environment is intact; an EXIT trap copies the refreshed file back
to the PVC whether the run succeeds or fails.
This was found while recovering from Audible refusing content licenses
for this account. Re-registering the account fixed the refusals; without
this change the fresh registration would have been discarded on the next
run exactly like every refresh before it.
Also corrects the init container comment, which claimed the sentinel
preserved token refreshes. It preserves the DB; it never preserved the
tokens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CronJob ran `rmcrackan/libation:latest` with imagePullPolicy Always
but was stuck on 13.7.11 (2026-08-24). Spegel answers containerd's tag
resolution from its peer cache -- a 200 from the mirror is a hit, so
upstream is never consulted and the tag never advances. Verified: the
pod's imageID was sha256:946177d3 (= tag 13.7.11) while Docker Hub's
:latest pointed at sha256:6f9d8ba1 (= 14.2.1), and querying a Spegel
pod's registry port directly returned the stale digest for that tag.
Audible changed device registration in early September; upstream shipped
an experimental fix in 14.1 and the real fix in 14.2, so the stale image
predates both. Since 2026-09-15 three newly purchased titles fail with
"Content license denied" and the library has not grown.
A digest reference is immune to the mirror's tag cache. Note that the
upstream maintainer's remedy also requires removing and re-adding the
Audible account with an interactive login, which this change does not
and cannot perform.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measuring VLAN 40 for the traffic-class split turned up a share of the
management segment that is not management: the node's Ceph client
talking to OSDs on the other two nodes, sourced from the node address
on mgmt0.
Ceph sends every client read to the acting primary of the PG, which
with three OSDs is remote two times out of three. Read affinity has the
client read from the replica in its own CRUSH location instead.
The preconditions hold here. replicapool and cephfs/data0 are size 3
over failureDomain host across three nodes, so every object has a copy
on every node. The OSDs are created under zone buckets
(root=default host=k8s-N zone=<name>) and the nodes carry
topology.kubernetes.io/zone, which is in Rook's default
crushLocationLabels. Ceph is v20.2.4 and the kernel 6.18, against a
documented minimum of 5.8.
Writes are unaffected: they always go to the primary, and the
replication behind them already rides the VLAN 49 cluster network.
cephfs-ci is size 2, so a local replica is not guaranteed there and the
client falls back to the primary.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rolling out #857 took cross-node pod traffic down, and cluster DNS with it, until it was reverted. The table was incomplete: the node's own pod CIDR was originated as a blackhole so it could be advertised without being installed, and then filtered out of the kernel export.
The rule matches the SOURCE, and a reply from a pod on another node is forwarded traffic with a pod source — so it matches too. Its destination is a pod on this node, the table had no route for that, and it took the default and left for the UDM. The comment claiming such traffic would fall through to main was wrong: nothing falls through once a default matches.
Originating it via cilium_host instead makes it both advertisable and correct in the table, and the export filter that existed only to hide the blackhole goes with it.
Diagnosed by narrowing the rule to a single pod address, where the same path worked because the return packet no longer matched: 404 with src <pod>/32, 000 with src 10.244.0.0/16, 404 again once 10.244.0.0/24 dev cilium_host was added to table 47 by hand. bird -p parses the corrected config inside the image, exit 0.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule stack grew a carve-out layer because table 47 held nothing but a default route: it matched everything, so every cluster-internal destination had to be excluded by a rule naming its prefix. Those exclusions were wrong twice over — too broad, since fd00::/8 and 10.0.0.0/8 also caught every LAN client and sent its reply back over mgmt0, and unnecessary, since the networks in question are directly attached and a connected route already beats a default.
So the table gets the connected routes and decides on its own. bird's direct protocol generates them from the interfaces rather than from a list, alongside the peers' pod CIDRs it already learns and the egress default it now originates. One rule per family remains, on the pod source prefix; 980/981, 990/991 and table 48 are gone.
bird owns the table outright, including the default. That is what keeps the failure benign: if the daemon stops the table empties, the rule finds nothing and falls through to main, so pod traffic returns to VLAN 40 unseparated rather than misrouted. A default declared in the Talos config would have outlived the specific routes and sent pod traffic to the UDM.
The ingress gateway moves from a route in each node patch to one node annotation in the site patch, which is now the only place it is stated.
Verified: bird -p parses the rendered config inside the image, exit 0; the templates render GW6 fd02:47::1 and GW4 10.2.47.1 from the live k8s-1 object; kubectl kustomize builds.
Refs #576
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# The UDM on the ingress segment. Published here rather than as a route
# because bird owns table 47 and originates the default from it; this is the
# one place the address is stated.
nodeAnnotations:
net.renner.uno/egress-gw-v6:fd02:47::1
net.renner.uno/egress-gw-v4:10.2.47.1
---
# Pod egress leaves over ingress0. Matching on the pod source prefix is what
# keeps the node's own traffic out of it: the node IP, the API VIP and apid all
# answer on mgmt0, and a reply that left over ingress0 would carry an mgmt0
# source onto VLAN 47.
# All pod traffic goes to table 47, and the table decides — longest prefix
# match, no exceptions to carve out. bird fills it with the node's attached
# networks, the peers' pod CIDRs and the egress default, so a packet to a node
# on the mgmt segment takes the connected route, a packet to a pod on another
# node takes pod0, and anything else takes the default over ingress0.
#
# The routing decision runs before Cilium's masquerade hook, so the rule sees
# the pod address and the masquerade then rewrites it to ingress0's.
# Matching the source is what keeps the node's own traffic out: it answers on
# mgmt0, and a reply leaving over ingress0 would carry an mgmt0 source onto
# VLAN 47. The routing decision runs before Cilium's masquerade hook, so the
# rule sees the pod address and the masquerade then rewrites it.
#
# Table 47 holds nothing but the two default routes, so the destination rules
# below have to run first: without them a pod's packet to a service or to a pod
# on another node also matches the source rule, finds only the default, and
# leaves for the UDM. Measured — in-cluster DNS stopped resolving.
#
# They send everything inside the cluster and the local networks back to main.
# The prefixes are Cilium's ipv4/ipv6NativeRoutingCIDR, which already mark what
# is not foreign traffic.
# If bird stops, table 47 empties, the lookup misses and the rule falls through
# to main — pod traffic returns to VLAN 40, unseparated rather than misrouted.
#
# Priorities are below main (32766) and clear of the rule Cilium installs at 9.
# The two families are separate rule tables, so the priorities need not match;
# the document name is the priority and must be unique.
# Pod to pod between nodes goes over pod0, via table 48, which the bird mesh
# fills with one route per peer pod CIDR and the peer's pod0 address as the
# next hop. Ahead of 990/991 so it wins for that traffic, and matching source
# AND destination so nothing else lands there: the node's own traffic to a pod
# would otherwise leave with a pod0 source, which CiliumNode does not list, so
# Cilium would classify it as world rather than host.
#
# Traffic to a pod on this node falls through — table 48 holds only the peers'
# CIDRs, so the lookup misses and 990/991 send it to main and cilium_host.
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"980"
src:fd10:244::/48
dst:fd10:244::/48
table:"48"
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"981"
src:10.244.0.0/16
dst:10.244.0.0/16
table:"48"
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"990"
dst:fd00::/8
table:main
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"991"
dst:10.0.0.0/8
table:main
---
apiVersion:v1alpha1
kind:RoutingRuleConfig
name:"1000"
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
security-bot
k.renner