Compare commits
20
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  claude-botandClaude Opus 5
|
9a521568c0 | ||
 security-bot
|
3b3ed4fbc4 | ||
 k.renner
|
c40466a3e6 | ||
|
k.renner
|
ff0b3434c1 | ||
|
claude-botandClaude Opus 5
|
2cfebc2f8b | ||
|
claude-botandClaude Opus 5
|
3481e43032 | ||
|
security-bot
|
98475a47e2 | ||
|
security-bot
|
6f20955e3b | ||
|
claude-botandClaude Opus 5
|
92ac021931 | ||
|
claude-botandClaude Opus 5
|
5bbca6a162 | ||
|
k.renner
|
edb07a344e | ||
|
claude-botandClaude Opus 5
|
260f548e67 | ||
|
k.renner
|
ef6bc24491 | ||
|
claude-botandClaude Opus 5
|
779f557624 | ||
|
k.renner
|
1fbefb3c39 | ||
|
claude-botandClaude Opus 5
|
f97aa077f0 | ||
|
k.renner
|
8aa662e997 | ||
|
security-bot
|
688f84dc29 | ||
|
k.renner
|
6335954d6c | ||
|
claude-botandClaude Opus 5
|
f0701370db |
@@ -0,0 +1,117 @@
|
||||
# Traffic classes on the k8s nodes
|
||||
|
||||
Issue #576. Three of the four classes are separated; management is not.
|
||||
|
||||
## Where each class runs
|
||||
|
||||
| class | link | mechanism |
|
||||
|---|---|---|
|
||||
| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
|
||||
| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
|
||||
| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
|
||||
| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
|
||||
| management | `mgmt0`, VLAN 40 | not started |
|
||||
|
||||
Hard constraint, set by the owner: inter-node and pod traffic never traverses
|
||||
the UDM. Everything above stays on the 10G switch.
|
||||
|
||||
## Why policy routing rather than moving the node IP
|
||||
|
||||
Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
|
||||
and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
|
||||
`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
|
||||
Both were established by test (#837, #838), not by reading.
|
||||
|
||||
So the InternalIP decides where pod traffic goes — unless the lookup happens in
|
||||
a different table. The rules do that, and nothing about the node's identity
|
||||
moves. The routing decision runs before Cilium's masquerade hook, so a rule
|
||||
matching the pod source sees the pod address, and the masquerade then rewrites
|
||||
it to the egress link's.
|
||||
|
||||
## Rule order
|
||||
|
||||
Lower priority runs first.
|
||||
|
||||
| prio | match | table |
|
||||
|---|---|---|
|
||||
| 980 | src pod, dst pod | 48 |
|
||||
| 981 | src pod, dst pod (v4) | 48 |
|
||||
| 990 | dst `fd00::/8` | main |
|
||||
| 991 | dst `10.0.0.0/8` | main |
|
||||
| 1000 | src pod | 47 |
|
||||
| 1001 | src pod (v4) | 47 |
|
||||
|
||||
980/981 match source **and** destination. Source alone would also catch the
|
||||
node's own traffic to a pod, which would then leave with a `pod0` source — an
|
||||
address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
|
||||
`world` rather than `host`.
|
||||
|
||||
990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
|
||||
cluster-internal traffic out of table 47. Without them a pod's packet to a
|
||||
service matches the source rule, finds only a default route, and leaves for the
|
||||
UDM — measured, in-cluster DNS stopped resolving.
|
||||
|
||||
Traffic to a pod on the same node falls through on its own: table 48 holds only
|
||||
the peers' CIDRs, so the lookup misses and 990/991 send it to main.
|
||||
|
||||
## Where each table comes from
|
||||
|
||||
- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
|
||||
The IPv6 gateway is the UDM's configured address on the segment, not the
|
||||
RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
|
||||
default does not land in main.
|
||||
- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
|
||||
per peer pod CIDR with the peer's `pod0` address as next hop.
|
||||
|
||||
## The bird mesh
|
||||
|
||||
`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
|
||||
and installs what it learns into table 48. Cilium's routes in main are
|
||||
untouched, which is why the two coexist.
|
||||
|
||||
No address is written into this repo twice. Each node publishes its `pod0`
|
||||
addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
|
||||
pod CIDRs come from the Node objects. An init container renders both from the
|
||||
API with `kubectl -o go-template-file`, excludes the node itself with a field
|
||||
selector, and ends with `bird -p` so BIRD's own parser gates startup.
|
||||
|
||||
The node's own CIDR is **originated**, not read back from the kernel. Reading
|
||||
it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
|
||||
rejected outright: its next hop sits inside the prefix it describes, which KRT
|
||||
refuses with `strange next-hop`. The IPv6 route is a device route and passed,
|
||||
so the failure was silent on one family only.
|
||||
|
||||
## Failure behaviour
|
||||
|
||||
Benign at every layer. An empty or absent table 48 makes 980/981 fall through
|
||||
to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
|
||||
holds if the DaemonSet cannot start, which is the case during a cold rebuild
|
||||
until Gitea serves its image.
|
||||
|
||||
## Management, the remaining class
|
||||
|
||||
Not started. The levers differ per service and none of them is the node IP:
|
||||
|
||||
| service | can leave the node IP? | lever |
|
||||
|---|---|---|
|
||||
| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
|
||||
| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
|
||||
| Talos API | n/a | `apid` answers on every node address; a firewall question |
|
||||
| kubelet | no | it *is* the node IP |
|
||||
|
||||
Two approaches were rejected by the owner and are not to be proposed again:
|
||||
moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
|
||||
cluster VLAN while moving etcd and the API VIP to a new management VLAN.
|
||||
Replacing the CNI (#839, Antrea) was shelved.
|
||||
|
||||
Note that the reasoning which produced that list also concluded pod↔pod was
|
||||
blocked, which policy routing disproved. Treat it as a record of decisions, not
|
||||
as an exhaustive analysis.
|
||||
|
||||
## Sources
|
||||
|
||||
Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
|
||||
`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
|
||||
`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
|
||||
`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
|
||||
#855; closed unmerged: #830, #838, #839.
|
||||
@@ -1,194 +0,0 @@
|
||||
# Proposal — separate the four traffic classes onto their own VLANs
|
||||
|
||||
Status: draft. The question this started with is **answered** — from the Cilium
|
||||
source and flag reference, not from a test — and the answer changes the shape
|
||||
of the proposal rather than confirming it.
|
||||
|
||||
Supersedes the framing in issue #576, which described this as "give node
|
||||
management its own NIC, and Ceph public its own VLAN". That was two of four
|
||||
classes and one of them turned out already solved.
|
||||
|
||||
## The four classes
|
||||
|
||||
| class | what it is | where it goes today |
|
||||
|---|---|---|
|
||||
| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
|
||||
| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
|
||||
| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
|
||||
| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
|
||||
|
||||
Three of four share one interface. That is not a design decision — until the
|
||||
bare-metal conversion there was one NIC to work with.
|
||||
|
||||
Ceph client I/O living on the pod network is correct rather than a leftover:
|
||||
that traffic *is* cluster-internal. Ceph is done.
|
||||
|
||||
## Principle
|
||||
|
||||
**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
|
||||
separate, changeable decision.**
|
||||
|
||||
Defining classes by VLAN means the separation survives recabling, port failures
|
||||
and capacity changes, and it does not require one port per class. Defining them
|
||||
by interface would freeze a capacity decision into the architecture.
|
||||
|
||||
## The finding that shapes everything else
|
||||
|
||||
The original open question was whether `nodePort.directRoutingDevice` moves the
|
||||
next-hop of Cilium's direct routes, or only selects an interface. It is
|
||||
answerable from the source, and the answer is: **only the interface.**
|
||||
|
||||
`installDirectRoute` takes the next-hop as a parameter:
|
||||
|
||||
```go
|
||||
func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
|
||||
skipUnreachable bool) (routeSpec *netlink.Route, err error)
|
||||
```
|
||||
|
||||
and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
|
||||
in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
|
||||
same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
|
||||
|
||||
`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
|
||||
an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
|
||||
`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
|
||||
cluster there is no path by which Cilium's node IP differs from the kubelet's
|
||||
`InternalIP`.
|
||||
|
||||
`--direct-routing-device` is documented as "Device name used to connect nodes in
|
||||
direct routing mode" — a device, not an address.
|
||||
|
||||
**So cluster-internal traffic is welded to the node IP, and the node IP is the
|
||||
kubelet InternalIP.** Two consequences:
|
||||
|
||||
- The class cannot be moved off whatever VLAN the node IP lives on.
|
||||
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
|
||||
from the node IP; native mode uses it as the direct-route next-hop. Native
|
||||
routing buys nothing here, so the disruptive, non-rolling step this proposal
|
||||
previously contained is simply gone.
|
||||
|
||||
## Consequence: draw the line the other way
|
||||
|
||||
If cluster-internal cannot leave the node IP, then separation has to come from
|
||||
moving *management services* off it — and each one has to be checked
|
||||
individually, because they do not share a mechanism.
|
||||
|
||||
| management service | can it leave the node IP? | lever |
|
||||
|---|---|---|
|
||||
| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
|
||||
| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
|
||||
| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
|
||||
| kubelet | **no** | it *is* the node IP, by definition |
|
||||
|
||||
So four VLANs are reachable, but the boundary does not fall where the class
|
||||
names suggest: **kubelet stays with cluster-internal.** That is the honest
|
||||
result, and it is worth stating plainly rather than filing kubelet under
|
||||
management and hoping.
|
||||
|
||||
The remaining classes keep the levers established earlier:
|
||||
|
||||
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
|
||||
For eBGP the session source is the next-hop, so pinning the session pins
|
||||
where return traffic arrives. The session to the UDM has been running for
|
||||
months.
|
||||
- **cluster-external egress** — eBPF masquerading follows the output device, so
|
||||
this is a default-route decision plus `enable-masquerade-to-route-source`
|
||||
(today `false`).
|
||||
- **Ceph replication** — Multus macvlan on VLAN 49, done.
|
||||
|
||||
## The general escape hatch
|
||||
|
||||
Ceph is separated because its OSDs got a second interface and stopped using the
|
||||
pod network for replication. That is the *only* mechanism that fully detaches a
|
||||
workload from the node IP, and it generalises: anything that must not ride the
|
||||
node IP gets its own `NetworkAttachmentDefinition`.
|
||||
|
||||
It is also expensive per workload — an attachment, an IPAM pool, a pod spec
|
||||
change. Worth it for Ceph replication. Not worth it for east-west traffic in
|
||||
general.
|
||||
|
||||
## Recommended mapping
|
||||
|
||||
Two ways to arrange this, and they differ only in which VLAN gets relabelled:
|
||||
|
||||
**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
|
||||
and Talos API. Cleanest match to the class names, but it changes every node's
|
||||
identity, which is the one address the whole cluster is administered through.
|
||||
|
||||
**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
|
||||
VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
|
||||
Identical end state, and no node ever changes its InternalIP.
|
||||
|
||||
Under B:
|
||||
|
||||
| VLAN | carries |
|
||||
|---|---|
|
||||
| 49 | Ceph replication |
|
||||
| 40 (relabelled *cluster*) | pod east-west, kubelet |
|
||||
| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
|
||||
| new *external* | LoadBalancer ingress, egress default route |
|
||||
|
||||
The property the user asked for — management carries only management traffic —
|
||||
holds under B. The label moves; the property is what matters.
|
||||
|
||||
## Migration shape
|
||||
|
||||
Every step is rolling and reversible. The routing-mode change is gone.
|
||||
|
||||
1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
|
||||
2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
|
||||
Still unused; this only proves the L2 works.
|
||||
3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
|
||||
waiting for a healthy quorum between each.
|
||||
4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
|
||||
`k8s.renner.uno` follows.
|
||||
5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
|
||||
Reversible by reverting the override; failure is visible immediately in
|
||||
`cilium-dbg shell -- bgp/peers`.
|
||||
6. Verify VLAN 40 by what is *absent* from it, not by what still works.
|
||||
|
||||
Step 3 is the one that deserves care — etcd is the only step where a mistake
|
||||
costs quorum rather than connectivity.
|
||||
|
||||
## What this does not buy
|
||||
|
||||
Management currently shares a 10 Gb link and is not contended. Nothing observed
|
||||
today is slow because of this. The case for the split is that traffic classes
|
||||
with different failure modes and security postures should not be
|
||||
indistinguishable — not that something is broken.
|
||||
|
||||
## Sources and measurements
|
||||
|
||||
Source, Cilium v1.20.1 (the running version):
|
||||
|
||||
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
|
||||
fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
|
||||
a next-hop that is not directly reachable.
|
||||
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
|
||||
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
|
||||
`--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
|
||||
Kubernetes Node.
|
||||
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
|
||||
to connect nodes in direct routing mode".
|
||||
|
||||
Talos:
|
||||
|
||||
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
|
||||
defaults to it. kube-apiserver and Talos API listen addresses are not
|
||||
covered there.
|
||||
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
|
||||
`10.2.40.60` with `link: mgmt0`.
|
||||
|
||||
Measured on the cluster:
|
||||
|
||||
- `apid` answered on a node address that is not the InternalIP, so the Talos
|
||||
API is not bound to one address.
|
||||
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
|
||||
InternalIP had changed since the last session, which is why nothing here
|
||||
names an address as fixed.
|
||||
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
|
||||
unset and no direct-routing device configured.
|
||||
|
||||
Not verified: whether the kube-apiserver's `--advertise-address` follows the
|
||||
node IP independently of the VIP. It does not affect the recommendation, since
|
||||
clients reach the API through the VIP either way.
|
||||
@@ -70,7 +70,7 @@ spec:
|
||||
mountPath: /ca
|
||||
containers:
|
||||
- name: audiobookshelf
|
||||
image: ghcr.io/advplyr/audiobookshelf:2.36.0 # {"$imagepolicy": "flux-system:audiobookshelf"}
|
||||
image: ghcr.io/advplyr/audiobookshelf:2.36.1 # {"$imagepolicy": "flux-system:audiobookshelf"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 80
|
||||
|
||||
@@ -40,7 +40,7 @@ metadata:
|
||||
# so this Service produces no public record — LDAP stays
|
||||
# LAN-internal, which matches the threat model of the cleartext
|
||||
# bind on 389.
|
||||
external-dns.alpha.kubernetes.io/hostname: ldap.renner.uno
|
||||
external-dns.kubernetes.io/hostname: ldap.renner.uno
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
@@ -18,7 +18,7 @@ metadata:
|
||||
lb-pool: bgp
|
||||
annotations:
|
||||
lbipam.cilium.io/ips: "10.2.99.87,fd99::87"
|
||||
external-dns.alpha.kubernetes.io/hostname: ha.renner.uno
|
||||
external-dns.kubernetes.io/hostname: ha.renner.uno
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
@@ -19,7 +19,7 @@ metadata:
|
||||
# LAN can connect with `mqtt.renner.uno` (matches the broker cert
|
||||
# SAN) instead of the bare IP. No Cloudflare label — kept off the
|
||||
# public DNS; UniFi serves a split-DNS answer for LAN clients.
|
||||
external-dns.alpha.kubernetes.io/hostname: mqtt.renner.uno
|
||||
external-dns.kubernetes.io/hostname: mqtt.renner.uno
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
@@ -42,9 +42,11 @@ spec:
|
||||
type: RuntimeDefault
|
||||
initContainers:
|
||||
# Seed the libation-config PVC from the bootstrap secret on
|
||||
# first run only — sentinel file gates re-runs so token
|
||||
# refreshes in AccountsSettings.json and DB updates from
|
||||
# subsequent liberate passes are preserved.
|
||||
# first run only — sentinel file gates re-runs so DB updates
|
||||
# from subsequent liberate passes are preserved. Refreshed
|
||||
# Audible tokens are preserved by the container args, not by
|
||||
# this sentinel: the entrypoint copies AccountsSettings.json
|
||||
# into ephemeral /config-internal rather than symlinking it.
|
||||
- name: seed-config
|
||||
image: busybox:1.37.0
|
||||
command: ["/bin/sh", "-eu", "-c"]
|
||||
@@ -66,6 +68,11 @@ spec:
|
||||
install -m 660 /bootstrap/AccountsSettings.json /libation-state/AccountsSettings.json
|
||||
install -m 660 /bootstrap/Settings.json /libation-state/Settings.json
|
||||
install -m 660 /bootstrap/LibationContext.db /libation-state/LibationContext.db
|
||||
# Without this key the seeded tokens cannot be decrypted:
|
||||
# TokenStorageMethod is Encrypted and the container has no
|
||||
# OS secret store, so a missing key file makes the
|
||||
# entrypoint mint a throwaway one on every run.
|
||||
install -m 660 /bootstrap/libation-master.key /libation-state/libation-master.key
|
||||
touch /libation-state/.bootstrapped
|
||||
ls -la /libation-state
|
||||
volumeMounts:
|
||||
@@ -85,7 +92,23 @@ spec:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: libation
|
||||
image: docker.io/rmcrackan/libation:latest
|
||||
# Pinned by digest: Spegel caches mutable tag resolutions, so
|
||||
# :latest stayed on 13.7.11 despite imagePullPolicy Always.
|
||||
image: docker.io/rmcrackan/libation:14.2.1@sha256:6f9d8ba19dc3803dd6ba295ab5a7abfb6ff3bc06286eebf4ecd8ccbe43245888
|
||||
# The entrypoint stages config into ephemeral /config-internal
|
||||
# by copy (only the DB is symlinked to the PVC), so Audible
|
||||
# tokens refreshed during a run are lost at container exit.
|
||||
# Passing args makes the entrypoint set up the environment and
|
||||
# then exec this instead of its own loop; the trap copies the
|
||||
# refreshed file back to the PVC on every exit path.
|
||||
args:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
trap 'cp /config-internal/AccountsSettings.json /libation-state/AccountsSettings.json' EXIT
|
||||
set -e
|
||||
/libation/LibationCli scan
|
||||
/libation/LibationCli liberate
|
||||
env:
|
||||
- name: LIBATION_CONFIG_DIR
|
||||
value: /libation-state
|
||||
|
||||
@@ -232,7 +232,7 @@ metadata:
|
||||
lb-pool: bgp
|
||||
annotations:
|
||||
io.cilium/lb-ipam-ips: "10.2.99.84,fd99::84"
|
||||
external-dns.alpha.kubernetes.io/hostname: syslog.renner.uno
|
||||
external-dns.kubernetes.io/hostname: syslog.renner.uno
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
@@ -43,7 +43,7 @@ spec:
|
||||
mountPath: /consume
|
||||
containers:
|
||||
- name: vsftpd
|
||||
image: gitea.renner.uno/k.renner/vsftpd:build.1784731469 # {"$imagepolicy": "flux-system:vsftpd"}
|
||||
image: gitea.renner.uno/k.renner/vsftpd:build.1789630783 # {"$imagepolicy": "flux-system:vsftpd"}
|
||||
ports:
|
||||
- name: ftp-ctrl
|
||||
containerPort: 21
|
||||
@@ -70,24 +70,19 @@ spec:
|
||||
- name: consume
|
||||
mountPath: /home/scanner
|
||||
subPath: scanner
|
||||
# FTP has no health verb, so the control port accepting a connection
|
||||
# is what can be asserted. It is enough for the failure that matters:
|
||||
# if vsftpd dies the scanner cannot deposit anything and nothing else
|
||||
# in the chain notices, because paperless only ever sees an empty
|
||||
# consume directory — which is indistinguishable from nobody having
|
||||
# scanned.
|
||||
#
|
||||
# Only the control port is probed. The passive range is negotiated
|
||||
# per transfer and is not listening at rest.
|
||||
# Asserts the control socket is listening without opening an FTP
|
||||
# connection: vsftpd 3.0.5 segfaults its listener on a fraction of
|
||||
# the connections it accepts. The passive range is negotiated per
|
||||
# transfer and is not listening at rest, so it is not probed.
|
||||
livenessProbe:
|
||||
tcpSocket:
|
||||
port: ftp-ctrl
|
||||
exec:
|
||||
command: ["sh", "-c", "netstat -lnt | grep -q ':21 '"]
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 5
|
||||
readinessProbe:
|
||||
tcpSocket:
|
||||
port: ftp-ctrl
|
||||
exec:
|
||||
command: ["sh", "-c", "netstat -lnt | grep -q ':21 '"]
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
@@ -124,7 +119,7 @@ metadata:
|
||||
# label — FTP must not be exposed via the cargo tunnel (clear-text
|
||||
# protocol, public exposure not appropriate). The brother scanner
|
||||
# uses this hostname instead of the LB IP.
|
||||
external-dns.alpha.kubernetes.io/hostname: ftp.paperless.renner.uno
|
||||
external-dns.kubernetes.io/hostname: ftp.paperless.renner.uno
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
@@ -25,18 +25,15 @@ resources:
|
||||
configMapGenerator:
|
||||
# vsftpd config for the ftp-scanner deployment. The content-hash name
|
||||
# suffix is intentionally LEFT ENABLED (unlike home-assistant's nginx
|
||||
# config): vsftpd templates pasv_address from PASV_ADDRESS at container
|
||||
# start, so a changed value must roll the pod. With the hash suffix,
|
||||
# editing any value here renames the generated ConfigMap, which mutates
|
||||
# the Deployment's pod template and triggers an automatic rolling
|
||||
# restart. Without it, an IP change (e.g. an LB re-allocation) silently
|
||||
# leaves the running pod advertising the stale PASV address, which
|
||||
# breaks classic-PASV clients (older Brother scanners) while EPSV
|
||||
# clients keep working.
|
||||
# config): vsftpd reads these into vsftpd.conf at container start, so a
|
||||
# changed value must roll the pod, which the renamed ConfigMap does.
|
||||
#
|
||||
# No PASV_ADDRESS: the LB address is declared once, on the Service
|
||||
# (io.cilium/lb-ipam-ips). The image default 0.0.0.0 tells PASV clients
|
||||
# to reuse the control-connection address.
|
||||
- name: ftp-scanner-config
|
||||
namespace: paperless
|
||||
literals:
|
||||
- PASV_ADDRESS=10.2.99.82
|
||||
- PASV_MIN_PORT=21100
|
||||
- PASV_MAX_PORT=21104
|
||||
- FTP_USER=scanner
|
||||
|
||||
@@ -145,7 +145,7 @@ metadata:
|
||||
lb-pool: bgp
|
||||
annotations:
|
||||
io.cilium/lb-ipam-ips: "10.2.99.85,fd99::85"
|
||||
external-dns.alpha.kubernetes.io/hostname: syncthing.paperless.renner.uno
|
||||
external-dns.kubernetes.io/hostname: syncthing.paperless.renner.uno
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
@@ -134,7 +134,7 @@ spec:
|
||||
# is a device-bricking class of risk. Bumps are a reviewed, manual
|
||||
# act — see kubernetes/infrastructure/image-automation/README.md.
|
||||
- name: terminus
|
||||
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
image: ghcr.io/usetrmnl/terminus:0.74.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 2345
|
||||
@@ -197,7 +197,7 @@ spec:
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
- name: sidekiq
|
||||
image: ghcr.io/usetrmnl/terminus:0.73.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
image: ghcr.io/usetrmnl/terminus:0.74.0 # {"$imagepolicy": "flux-system:terminus"}
|
||||
command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
|
||||
@@ -21,7 +21,7 @@ spec:
|
||||
# Service; nothing on this path needs the host's network namespace.
|
||||
containers:
|
||||
- name: zone-configurator
|
||||
image: everythingsmarthome/everything-presence-mmwave-configurator:2.2.4 # {"$imagepolicy": "flux-system:mmwave-configurator"}
|
||||
image: everythingsmarthome/everything-presence-mmwave-configurator:2.2.5 # {"$imagepolicy": "flux-system:mmwave-configurator"}
|
||||
env:
|
||||
- name: HA_BASE_URL
|
||||
value: "http://home-assistant:80"
|
||||
|
||||
@@ -610,8 +610,8 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
|
||||
# Constraints on the choice: cpx11/cpx21 are US-only and cax* are ARM, which
|
||||
# would not match the amd64 schematic this cluster installs. That leaves the
|
||||
# cx*/cpx*2 families in the German locations.
|
||||
# Today's traffic-class split, not the target one in
|
||||
# PROPOSAL-traffic-class-separation.md: the node address carries management,
|
||||
# Two segments only, unlike the homelab's split in
|
||||
# NETWORK-traffic-classes.md: here the node address carries management,
|
||||
# cluster-internal and external alike, and Ceph replication has its own.
|
||||
#
|
||||
# The order is load-bearing: it decides which kernel interface each segment
|
||||
|
||||
@@ -39,37 +39,33 @@ protocol static egress6 {
|
||||
route ::/0 via GW6;
|
||||
}
|
||||
|
||||
# This node's own pod CIDR, originated rather than read back out of the kernel.
|
||||
# Reading it back made BIRD's view depend on the shape of Cilium's route, and
|
||||
# the IPv4 one is rejected outright: its next hop sits inside the prefix it
|
||||
# describes, which KRT refuses with "strange next-hop".
|
||||
# This node's own pod CIDR, via the interface Cilium serves it on. Originated
|
||||
# rather than read back out of the kernel: reading it back made BIRD's view
|
||||
# depend on the shape of Cilium's route, and the IPv4 one is rejected outright,
|
||||
# its next hop sitting inside the prefix it describes.
|
||||
#
|
||||
# blackhole because it is only ever advertised — it is filtered out below.
|
||||
# It belongs in the table, not just in the BGP announcement. A reply from a pod
|
||||
# on another node is forwarded traffic with a pod source, so the rule matches it
|
||||
# too; without this route it would find only the default and leave for the UDM.
|
||||
# Measured — cross-node pod traffic, and with it cluster DNS, stopped.
|
||||
protocol static own4 {
|
||||
ipv4 { table master4; };
|
||||
route OWN_POD4 blackhole;
|
||||
route OWN_POD4 via "cilium_host";
|
||||
}
|
||||
protocol static own6 {
|
||||
ipv6 { table master6; };
|
||||
route OWN_POD6 blackhole;
|
||||
}
|
||||
|
||||
# Everything except this node's own CIDR, which cilium_host already serves in
|
||||
# main and which is a blackhole here.
|
||||
filter to_pod_table {
|
||||
if source = RTS_STATIC && dest = RTD_BLACKHOLE then reject;
|
||||
accept;
|
||||
route OWN_POD6 via "cilium_host";
|
||||
}
|
||||
|
||||
protocol kernel k4 {
|
||||
kernel table 47;
|
||||
metric 0;
|
||||
ipv4 { table master4; import none; export filter to_pod_table; };
|
||||
ipv4 { table master4; import none; export all; };
|
||||
}
|
||||
protocol kernel k6 {
|
||||
kernel table 47;
|
||||
metric 0;
|
||||
ipv6 { table master6; import none; export filter to_pod_table; };
|
||||
ipv6 { table master6; import none; export all; };
|
||||
}
|
||||
|
||||
# iBGP over pod0. 64514 keeps clear of Cilium's 64513, which peers with the UDM
|
||||
|
||||
@@ -126,6 +126,16 @@ spec:
|
||||
addressRanges:
|
||||
cluster:
|
||||
- 10.99.0.0/29
|
||||
# Reads go to the OSD in the client's own CRUSH location. crushLocationLabels
|
||||
# is unset, so Rook's default list applies; of those the nodes carry
|
||||
# topology.kubernetes.io/zone, one zone per node, matching the zone buckets
|
||||
# the OSDs are created under.
|
||||
#
|
||||
# Passed as a krbd/mount option at map time, so a volume keeps its current
|
||||
# behaviour until it is remounted.
|
||||
csi:
|
||||
readAffinity:
|
||||
enabled: true
|
||||
# Nodes listed explicitly, not useAllNodes: under useAllNodes the operator
|
||||
# re-provisions an OSD on any node whose device still carries a BlueStore
|
||||
# signature, so purging an OSD before pulling its drive is impossible — it
|
||||
|
||||
Reference in New Issue
Block a user