Vier Klassen, ausgearbeitet: Ceph-Replikation, cluster-intern, cluster-extern, Management. Jede bekommt ein eigenes VLAN; auf welchem physischen Interface das landet, ist eine getrennte und änderbare Entscheidung.
Ceph ist fertig — und zwar vollständig, nicht halb: der Client-I/O gehört auf das Pod-Netz, weil dieser Traffic cluster-intern ist, nicht Ceph-Traffic. Die Replikation liegt auf VLAN 49.
Drei der vier Klassen lassen sich mit dokumentierten Hebeln trennen:
Management bewegt sich nicht. Die InternalIP definiert die Klasse, und validSubnets: 10.2.40.0/24 steht bereits richtig. Alles andere zieht weg — das ist auch die risikoärmere Richtung, weil die Adresse, über die der Cluster administriert wird, nie wechselt.
Ingress über CiliumBGPNodeConfigOverride…peers[].localAddress, auf einer BGP-Session, die seit Monaten läuft.
Egress folgt unter eBPF-Masquerading ohnehin dem Ausgabegerät.
Cluster-intern löst sich nicht. Im Tunnel-Modus stammt die VXLAN-Quelle von der Node-IP, ohne dokumentierte Alternative. Native Routing wäre der Ausweg und die Voraussetzung ist erfüllt — BGP annonciert die Pod-CIDRs die ganze Zeit schon an die UDM. Aber ob nodePort.directRoutingDevice den Next-Hop mitnimmt oder nur das Interface wählt, steht in keiner der drei geprüften Cilium-Seiten.
Deshalb endet das Dokument mit einer Frage statt mit einem Plan, und der nächste Schritt ist eine Sandbox aus zwei Talos-VMs, kein Kabel. Fällt die Antwort ungünstig aus, ändert sich der Entwurf, nicht die Verkabelung.
Was gemessen wurde, steht unten im Dokument getrennt von dem, was angenommen ist — inklusive der Beobachtung, die nicht zählt: dass Cilium beim Hinzufügen einer zweiten Adresse (Direct Routing) auf mgmt0 ließ, sagt nichts aus, solange der Cluster im Tunnel-Modus läuft.
Ehrlich zum Nutzen: Management teilt sich heute einen 10-Gb-Link und ist nicht ausgelastet. Nichts ist deswegen langsam. Das Argument ist Trennung von Fehler- und Sicherheitsdomänen, nicht Performance — und genau deshalb muss Schritt 5 (Umstellung auf native Routing, der einzige nicht-rollende Schritt) nicht schnell passieren.
Ersetzt die Rahmung in #576, die zwei der vier Klassen benannte, davon eine bereits gelöste.
Vier Klassen, ausgearbeitet: **Ceph-Replikation**, **cluster-intern**, **cluster-extern**, **Management**. Jede bekommt ein eigenes VLAN; auf welchem physischen Interface das landet, ist eine getrennte und änderbare Entscheidung.
Ceph ist fertig — und zwar vollständig, nicht halb: der Client-I/O gehört auf das Pod-Netz, weil dieser Traffic *cluster-intern* ist, nicht Ceph-Traffic. Die Replikation liegt auf VLAN 49.
Drei der vier Klassen lassen sich mit dokumentierten Hebeln trennen:
- **Management** bewegt sich nicht. Die InternalIP definiert die Klasse, und `validSubnets: 10.2.40.0/24` steht bereits richtig. Alles andere zieht weg — das ist auch die risikoärmere Richtung, weil die Adresse, über die der Cluster administriert wird, nie wechselt.
- **Ingress** über `CiliumBGPNodeConfigOverride…peers[].localAddress`, auf einer BGP-Session, die seit Monaten läuft.
- **Egress** folgt unter eBPF-Masquerading ohnehin dem Ausgabegerät.
**Cluster-intern löst sich nicht.** Im Tunnel-Modus stammt die VXLAN-Quelle von der Node-IP, ohne dokumentierte Alternative. Native Routing wäre der Ausweg und die Voraussetzung ist erfüllt — BGP annonciert die Pod-CIDRs die ganze Zeit schon an die UDM. Aber ob `nodePort.directRoutingDevice` den Next-Hop mitnimmt oder nur das Interface wählt, steht in keiner der drei geprüften Cilium-Seiten.
Deshalb endet das Dokument mit einer Frage statt mit einem Plan, und der nächste Schritt ist eine Sandbox aus zwei Talos-VMs, kein Kabel. Fällt die Antwort ungünstig aus, ändert sich der Entwurf, nicht die Verkabelung.
Was gemessen wurde, steht unten im Dokument getrennt von dem, was angenommen ist — inklusive der Beobachtung, die *nicht* zählt: dass Cilium beim Hinzufügen einer zweiten Adresse `(Direct Routing)` auf `mgmt0` ließ, sagt nichts aus, solange der Cluster im Tunnel-Modus läuft.
Ehrlich zum Nutzen: Management teilt sich heute einen 10-Gb-Link und ist nicht ausgelastet. Nichts ist deswegen langsam. Das Argument ist Trennung von Fehler- und Sicherheitsdomänen, nicht Performance — und genau deshalb muss Schritt 5 (Umstellung auf native Routing, der einzige nicht-rollende Schritt) nicht schnell passieren.
Ersetzt die Rahmung in #576, die zwei der vier Klassen benannte, davon eine bereits gelöste.
Issue #576 framed this as "give node management its own NIC, and Ceph public
its own VLAN". That is two of four classes, and one of them turned out to be
solved already — Ceph client I/O riding the pod network is correct, since that
traffic is cluster-internal.
The classes are Ceph replication (done, VLAN 49), cluster-internal,
cluster-external and management. Three of the four share mgmt0, which was never
a decision: until the bare-metal conversion there was one NIC.
Each class gets a VLAN; which interface a VLAN lands on stays a separate and
changeable decision. That ordering means the separation survives recabling and
does not need one port per class.
Three of the four are controllable with documented levers. Management is the
InternalIP and does not move — everything else moves away from it, which is
also the direction that never changes the address the cluster is administered
through. Ingress is CiliumBGPNodeConfigOverride's per-peer localAddress, on a
BGP session that has been running for months. Egress follows the output device
under eBPF masquerading.
Cluster-internal is the one that does not resolve. In tunnel mode VXLAN sources
from the node IP with no documented alternative. Native routing is the way out
and its prerequisite is already met — BGP has been advertising the pod CIDRs to
the UDM all along — but whether nodePort.directRoutingDevice moves the next-hop
address or only the interface is undocumented across the three Cilium pages
checked.
So the proposal ends on a question rather than a plan, and the next step is a
two-interface Talos sandbox, not a cable. If the answer is unfavourable the
design changes, not the wiring.
Records what was measured rather than assumed: the BGP session state, that
Cilium's direct-routing device follows the InternalIP (adding a second address
elsewhere left it on mgmt0), that the override CRD exposes localAddress, and
where Ceph's two address families actually sit.
Also states what the split does not buy. Management shares a 10 Gb link and is
not contended; nothing observed is slow because of this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 6a50ab1daa86fc2f9b55b2f43af8166826e5bc2c.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR proposes a framework for separating four traffic classes (Ceph replication, cluster-internal, cluster-external, management) onto their own VLANs. The proposal is well-structured, honestly identifies a key unresolved technical prerequisite, and recommends testing before any production changes. No blocking issues; the review offers clarifications on the BGP configuration details and the control-plane changes needed to execute this plan.
🟡 Suggestions
1. Clarify BGP configuration strategy for ingress separation (line 73–89)
The proposal suggests using CiliumBGPNodeConfigOverride...peers[].localAddress to control which node address the BGP session sources from, to separate ingress traffic. However, the current production BGP setup uses CiliumBGPClusterConfig (cluster-level, symmetric across all nodes) rather than per-node overrides.
The comments in kubernetes/infrastructure/post-crds/cilium-bgp.yaml (line 31–32) note "Each node sources from its eth0 (10.2.40.x / fd40::) via the kernel routing decision; no explicit localAddress override needed." This suggests that for the current two-peer design, switching to per-node configuration with explicit localAddress overrides may be more complex than the proposal indicates—it could require migrating the entire BGP control-plane from cluster-wide to per-node configuration.
Recommendation: Clarify whether step 4 (separating cluster-external) requires switching from CiliumBGPClusterConfig to CiliumBGPNodeConfigOverride, or whether the ingress peer can instead be added to the existing cluster-wide config with a new localAddress field. The sandbox test should verify which approach works and whether they have different failure modes.
2. IPv6 considerations not addressed (line 122–134)
The proposal focuses on IPv4, but the configuration already has native routing CIDRs defined for IPv6 (ipv6-native-routing-cidr: "fd00::/8"). The migration steps should acknowledge whether the same traffic class separation applies to IPv6 peers and whether step 5 (native routing mode) requires addressing both families together or separately.
3. Missing details on directRoutingDevice behavior testing (line 120–142)
The open question is critical and correctly identified. The sandbox test should record:
What cilium-dbg ip route list outputs for direct routes (to see if next-hop changes)
Whether mixed routing (native routes on one VLAN, VXLAN encapsulation on another) is stable
Failure recovery: what happens when a direct route goes unreachable on the chosen device
This will help avoid surprises during step 5 if the answer is unfavorable.
4. Step 2 (new VLANs on switch) is not actually "free" (line 149–150)
The proposal claims "Steps 2 and 3 are free." However, step 2 requires switch reconfiguration, which has lead time, vendor dependencies, and a non-zero risk of L2 misconfiguration (e.g., wrong port trunks, wrong VLAN IDs). While reversible, it is not risk-free and deserves acknowledgment that it depends on access to switch configuration and validation procedures.
🟢 Nits
Line 21: "East-west pod traffic is VXLAN today" — minor: Cilium terminology typically refers to this as "tunnel mode" rather than just "VXLAN today," since VXLAN is the tunnel protocol but tunnel mode is the routing mode. Not wrong, just could be more precise.
Line 175: Ceph client I/O is correctly identified as pod-network traffic (front_addr on 10.244.x), but the writeup could emphasize that this is already solved by virtue of Ceph being in the pod network, not needing further work.
Tests
This is a proposal document, not testable code, so traditional testing does not apply. However, the proposal correctly prioritizes a sandbox test as step 1. Before production rollout, verify:
The sandbox confirms nodePort.directRoutingDevice behavior as specified in the open question
Both IPv4 and IPv6 routes behave predictably under the proposed direct-routing setup
Rolling node updates (to add new VLAN addresses and update Cilium config) do not disrupt cluster-internal traffic
BGP session failover works when a node's external VLAN address is briefly unavailable during reboot
Recommendation
Looks reasonable — no blocking issues; suggestions above are optional clarifications. The proposal is honest about what is and is not known, prioritizes testing over production deployment, and follows a reversible migration strategy. Proceed with the sandbox phase as described in step 1.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `6a50ab1daa86fc2f9b55b2f43af8166826e5bc2c`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR proposes a framework for separating four traffic classes (Ceph replication, cluster-internal, cluster-external, management) onto their own VLANs. The proposal is well-structured, honestly identifies a key unresolved technical prerequisite, and recommends testing before any production changes. No blocking issues; the review offers clarifications on the BGP configuration details and the control-plane changes needed to execute this plan.
## 🟡 Suggestions
**1. Clarify BGP configuration strategy for ingress separation (line 73–89)**
The proposal suggests using `CiliumBGPNodeConfigOverride...peers[].localAddress` to control which node address the BGP session sources from, to separate ingress traffic. However, the current production BGP setup uses `CiliumBGPClusterConfig` (cluster-level, symmetric across all nodes) rather than per-node overrides.
The comments in `kubernetes/infrastructure/post-crds/cilium-bgp.yaml` (line 31–32) note "Each node sources from its eth0 (10.2.40.x / fd40::<host>) via the kernel routing decision; no explicit localAddress override needed." This suggests that for the current two-peer design, switching to per-node configuration with explicit `localAddress` overrides may be more complex than the proposal indicates—it could require migrating the entire BGP control-plane from cluster-wide to per-node configuration.
**Recommendation**: Clarify whether step 4 (separating cluster-external) requires switching from `CiliumBGPClusterConfig` to `CiliumBGPNodeConfigOverride`, or whether the ingress peer can instead be added to the existing cluster-wide config with a new `localAddress` field. The sandbox test should verify which approach works and whether they have different failure modes.
**2. IPv6 considerations not addressed (line 122–134)**
The proposal focuses on IPv4, but the configuration already has native routing CIDRs defined for IPv6 (`ipv6-native-routing-cidr: "fd00::/8"`). The migration steps should acknowledge whether the same traffic class separation applies to IPv6 peers and whether step 5 (native routing mode) requires addressing both families together or separately.
**3. Missing details on `directRoutingDevice` behavior testing (line 120–142)**
The open question is critical and correctly identified. The sandbox test should record:
- What `cilium-dbg ip route list` outputs for direct routes (to see if next-hop changes)
- Whether mixed routing (native routes on one VLAN, VXLAN encapsulation on another) is stable
- Failure recovery: what happens when a direct route goes unreachable on the chosen device
This will help avoid surprises during step 5 if the answer is unfavorable.
**4. Step 2 (new VLANs on switch) is not actually "free" (line 149–150)**
The proposal claims "Steps 2 and 3 are free." However, step 2 requires switch reconfiguration, which has lead time, vendor dependencies, and a non-zero risk of L2 misconfiguration (e.g., wrong port trunks, wrong VLAN IDs). While reversible, it is not risk-free and deserves acknowledgment that it depends on access to switch configuration and validation procedures.
## 🟢 Nits
- Line 21: "East-west pod traffic is VXLAN today" — minor: Cilium terminology typically refers to this as "tunnel mode" rather than just "VXLAN today," since VXLAN is the tunnel protocol but tunnel mode is the routing mode. Not wrong, just could be more precise.
- Line 175: Ceph client I/O is correctly identified as pod-network traffic (front_addr on 10.244.x), but the writeup could emphasize that this is already solved by virtue of Ceph being in the pod network, not needing further work.
## Tests
This is a proposal document, not testable code, so traditional testing does not apply. However, the proposal correctly prioritizes a sandbox test as step 1. Before production rollout, verify:
- The sandbox confirms `nodePort.directRoutingDevice` behavior as specified in the open question
- Both IPv4 and IPv6 routes behave predictably under the proposed direct-routing setup
- Rolling node updates (to add new VLAN addresses and update Cilium config) do not disrupt cluster-internal traffic
- BGP session failover works when a node's external VLAN address is briefly unavailable during reboot
## Recommendation
Looks reasonable — no blocking issues; suggestions above are optional clarifications. The proposal is honest about what is and is not known, prioritizes testing over production deployment, and follows a reversible migration strategy. Proceed with the sandbox phase as described in step 1.
<!-- pr-review:sha=6a50ab1daa86fc2f9b55b2f43af8166826e5bc2c -->
The first draft ended on an open question and proposed a two-VM sandbox to
answer it. That was unnecessary: the question is answerable from the Cilium
source and the agent flag reference, and the answer is negative.
installDirectRoute takes its next-hop as a parameter, and nodeUpdate supplies
GetNodeIP(false), which prefers NodeInternalIP. NodeInternalIP comes from the
Kubernetes Node object; --ipv4-node looks like an override but initFromK8s()
runs after initFromConfig() and overwrites it. --direct-routing-device selects
a device, not an address. The same value feeds the VXLAN endpoint in tunnel
mode.
So cluster-internal traffic is welded to the kubelet InternalIP, and the routing
mode does not matter — tunnel sources the VXLAN from it, native uses it as the
next-hop. That deletes the one disruptive, non-rolling step the first draft
contained. Native routing buys nothing for this problem.
Since cluster-internal cannot be moved, the separation has to come from moving
management services off the node IP, and those do not share a mechanism, so each
is checked separately: etcd moves via cluster.etcd.advertisedSubnets, the
apiserver endpoint is already a VIP whose Layer2VIPConfig carries a link field,
the Talos API turned out to answer on every node address, and kubelet cannot
move because it is the node IP.
The result is four VLANs, but the boundary does not fall where the class names
suggest — kubelet stays with cluster-internal. Stated plainly rather than filed
under management and hoped over.
Recommends leaving the node IP where it is and relabelling VLAN 40 as the
cluster VLAN, rather than moving every node's identity to a new VLAN. Same end
state, and the address the cluster is administered through never changes.
Also records that the Ceph pattern is the only mechanism that fully detaches a
workload from the node IP, and that it is too expensive to generalise to
east-west traffic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nachgetragen — die offene Frage war keine, sie stand im Quellcode.
installDirectRoute bekommt den Next-Hop als Parameter, nodeUpdate liefert GetNodeIP(false), und das bevorzugt NodeInternalIP. Der Wert kommt aus dem Kubernetes-Node-Objekt; --ipv4-node sieht nach Override aus, aber initFromK8s() läuft nach initFromConfig() und überschreibt ihn wieder. --direct-routing-device wählt ein Device, keine Adresse.
Damit ist cluster-intern an die Kubelet-InternalIP geschweißt — und der Routing-Modus ist egal, weil Tunnel-Modus die VXLAN-Quelle aus derselben Variable nimmt. Native Routing bringt für dieses Problem nichts. Der einzige nicht-rollende Schritt aus dem ersten Entwurf fällt ersatzlos weg.
Also andersherum getrennt: nicht cluster-intern von der Node-IP wegziehen, sondern die Management-Dienste. Die teilen sich keinen Mechanismus, jeder einzeln geprüft:
Dienst
verschiebbar
Hebel
etcd
ja
cluster.etcd.advertisedSubnets
kube-apiserver
ja
ist schon ein VIP, Layer2VIPConfig.link
Talos API
entfällt
apid antwortet auf jeder Node-Adresse (gemessen)
kubelet
nein
ist die Node-IP
Vier VLANs sind erreichbar, aber die Grenze läuft nicht dort, wo die Klassennamen sie vermuten lassen: kubelet bleibt bei cluster-intern. Das steht so drin, statt es unter Management einzusortieren und zu hoffen.
Empfehlung: Node-IP liegen lassen und VLAN 40 zum Cluster-VLAN umetikettieren, statt jedem Node eine neue Identität zu geben. Gleiches Ergebnis, und die Adresse, über die administriert wird, wechselt nie.
Nebenbei aufgefallen: k8s-1 hat seit der letzten Sitzung eine andere InternalIP (.171 → .169), der Cluster ist gesund. Deshalb nennt das Dokument keine Adresse als fix.
Nachgetragen — die offene Frage war keine, sie stand im Quellcode.
`installDirectRoute` bekommt den Next-Hop als Parameter, `nodeUpdate` liefert `GetNodeIP(false)`, und das bevorzugt `NodeInternalIP`. Der Wert kommt aus dem Kubernetes-Node-Objekt; `--ipv4-node` sieht nach Override aus, aber `initFromK8s()` läuft nach `initFromConfig()` und überschreibt ihn wieder. `--direct-routing-device` wählt ein Device, keine Adresse.
Damit ist cluster-intern an die Kubelet-InternalIP geschweißt — und der Routing-Modus ist egal, weil Tunnel-Modus die VXLAN-Quelle aus derselben Variable nimmt. **Native Routing bringt für dieses Problem nichts.** Der einzige nicht-rollende Schritt aus dem ersten Entwurf fällt ersatzlos weg.
Also andersherum getrennt: nicht cluster-intern von der Node-IP wegziehen, sondern die Management-Dienste. Die teilen sich keinen Mechanismus, jeder einzeln geprüft:
| Dienst | verschiebbar | Hebel |
|---|---|---|
| etcd | ja | `cluster.etcd.advertisedSubnets` |
| kube-apiserver | ja | ist schon ein VIP, `Layer2VIPConfig.link` |
| Talos API | entfällt | `apid` antwortet auf jeder Node-Adresse (gemessen) |
| kubelet | nein | *ist* die Node-IP |
Vier VLANs sind erreichbar, aber die Grenze läuft nicht dort, wo die Klassennamen sie vermuten lassen: **kubelet bleibt bei cluster-intern.** Das steht so drin, statt es unter Management einzusortieren und zu hoffen.
Empfehlung: Node-IP liegen lassen und VLAN 40 zum Cluster-VLAN umetikettieren, statt jedem Node eine neue Identität zu geben. Gleiches Ergebnis, und die Adresse, über die administriert wird, wechselt nie.
Nebenbei aufgefallen: k8s-1 hat seit der letzten Sitzung eine andere InternalIP (`.171` → `.169`), der Cluster ist gesund. Deshalb nennt das Dokument keine Adresse als fix.
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 55ea98c035095ec8c87fe24e7032db045daed074.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR introduces a detailed technical proposal for separating cluster network traffic into four distinct VLANs: Ceph replication, cluster-internal, cluster-external, and management. The document provides a thorough analysis of Cilium/Talos architecture, identifies which traffic classes can be separated using existing configuration levers, and proposes a rolling, reversible migration path. The key insight—that cluster-internal traffic is bound to the node IP regardless of routing mode—reshapes the proposal into a more practical scope than originally framed in issue #576.
🟡 Suggestions
Line 62: The claim "In a Kubernetes cluster there is no path by which Cilium's node IP differs from the kubelet's InternalIP" is strong. Slightly more precise to say "with this particular Cilium configuration" or "in this cluster's setup", since downstream Cilium versions or different init orderings could theoretically diverge.
Line 90: "That is the honest result" is slightly informal for the technical tone of the document. Consider "This is the accurate result" or simply end the previous sentence with the technical finding.
Line 154: Step 6 says "Verify VLAN 40 by what is absent from it, not by what still works" — worth a sentence explaining what success looks like here (e.g., "confirm no management traffic appears on VLAN 40" or similar concrete metric).
Validation approach: The PR body (German text) mentions the next step is a sandbox test with two Talos VMs. Consider adding a brief "Next steps" section to the proposal itself stating this, so readers know the proposal is draft pending validation.
🟢 Nits
Line 45–74: Long paragraph; consider breaking the explanation of GetNodeIP precedence into a separate bullet or sub-paragraph for clarity.
Line 87: apid answered on a node address — consider hyperlink or fuller context on where this was measured (e.g., which node, command run).
Tests
This is a documentation proposal, not code. No tests needed. The approach of validating in a sandbox environment (two Talos VMs) before actual recabling is sensible given the disruptive nature of network changes.
Recommendation
Looks reasonable — no blocking issues; comments above are optional refinements to clarity and completeness.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `55ea98c035095ec8c87fe24e7032db045daed074`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR introduces a detailed technical proposal for separating cluster network traffic into four distinct VLANs: Ceph replication, cluster-internal, cluster-external, and management. The document provides a thorough analysis of Cilium/Talos architecture, identifies which traffic classes can be separated using existing configuration levers, and proposes a rolling, reversible migration path. The key insight—that cluster-internal traffic is bound to the node IP regardless of routing mode—reshapes the proposal into a more practical scope than originally framed in issue #576.
## 🟡 Suggestions
1. **Line 62**: The claim "In a Kubernetes cluster there is no path by which Cilium's node IP differs from the kubelet's `InternalIP`" is strong. Slightly more precise to say "with this particular Cilium configuration" or "in this cluster's setup", since downstream Cilium versions or different init orderings could theoretically diverge.
2. **Line 90**: "That is the honest result" is slightly informal for the technical tone of the document. Consider "This is the accurate result" or simply end the previous sentence with the technical finding.
3. **Line 154**: Step 6 says "Verify VLAN 40 by what is *absent* from it, not by what still works" — worth a sentence explaining what success looks like here (e.g., "confirm no management traffic appears on VLAN 40" or similar concrete metric).
4. **Validation approach**: The PR body (German text) mentions the next step is a sandbox test with two Talos VMs. Consider adding a brief "Next steps" section to the proposal itself stating this, so readers know the proposal is draft pending validation.
## 🟢 Nits
- Line 45–74: Long paragraph; consider breaking the explanation of `GetNodeIP` precedence into a separate bullet or sub-paragraph for clarity.
- Line 87: `apid` answered on a node address — consider hyperlink or fuller context on where this was measured (e.g., which node, command run).
## Tests
This is a documentation proposal, not code. No tests needed. The approach of validating in a sandbox environment (two Talos VMs) before actual recabling is sensible given the disruptive nature of network changes.
## Recommendation
Looks reasonable — no blocking issues; comments above are optional refinements to clarity and completeness.
<!-- pr-review:sha=55ea98c035095ec8c87fe24e7032db045daed074 -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Vier Klassen, ausgearbeitet: Ceph-Replikation, cluster-intern, cluster-extern, Management. Jede bekommt ein eigenes VLAN; auf welchem physischen Interface das landet, ist eine getrennte und änderbare Entscheidung.
Ceph ist fertig — und zwar vollständig, nicht halb: der Client-I/O gehört auf das Pod-Netz, weil dieser Traffic cluster-intern ist, nicht Ceph-Traffic. Die Replikation liegt auf VLAN 49.
Drei der vier Klassen lassen sich mit dokumentierten Hebeln trennen:
validSubnets: 10.2.40.0/24steht bereits richtig. Alles andere zieht weg — das ist auch die risikoärmere Richtung, weil die Adresse, über die der Cluster administriert wird, nie wechselt.CiliumBGPNodeConfigOverride…peers[].localAddress, auf einer BGP-Session, die seit Monaten läuft.Cluster-intern löst sich nicht. Im Tunnel-Modus stammt die VXLAN-Quelle von der Node-IP, ohne dokumentierte Alternative. Native Routing wäre der Ausweg und die Voraussetzung ist erfüllt — BGP annonciert die Pod-CIDRs die ganze Zeit schon an die UDM. Aber ob
nodePort.directRoutingDeviceden Next-Hop mitnimmt oder nur das Interface wählt, steht in keiner der drei geprüften Cilium-Seiten.Deshalb endet das Dokument mit einer Frage statt mit einem Plan, und der nächste Schritt ist eine Sandbox aus zwei Talos-VMs, kein Kabel. Fällt die Antwort ungünstig aus, ändert sich der Entwurf, nicht die Verkabelung.
Was gemessen wurde, steht unten im Dokument getrennt von dem, was angenommen ist — inklusive der Beobachtung, die nicht zählt: dass Cilium beim Hinzufügen einer zweiten Adresse
(Direct Routing)aufmgmt0ließ, sagt nichts aus, solange der Cluster im Tunnel-Modus läuft.Ehrlich zum Nutzen: Management teilt sich heute einen 10-Gb-Link und ist nicht ausgelastet. Nichts ist deswegen langsam. Das Argument ist Trennung von Fehler- und Sicherheitsdomänen, nicht Performance — und genau deshalb muss Schritt 5 (Umstellung auf native Routing, der einzige nicht-rollende Schritt) nicht schnell passieren.
Ersetzt die Rahmung in #576, die zwei der vier Klassen benannte, davon eine bereits gelöste.
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commit6a50ab1daa86fc2f9b55b2f43af8166826e5bc2c.Summary
This PR proposes a framework for separating four traffic classes (Ceph replication, cluster-internal, cluster-external, management) onto their own VLANs. The proposal is well-structured, honestly identifies a key unresolved technical prerequisite, and recommends testing before any production changes. No blocking issues; the review offers clarifications on the BGP configuration details and the control-plane changes needed to execute this plan.
🟡 Suggestions
1. Clarify BGP configuration strategy for ingress separation (line 73–89)
The proposal suggests using
CiliumBGPNodeConfigOverride...peers[].localAddressto control which node address the BGP session sources from, to separate ingress traffic. However, the current production BGP setup usesCiliumBGPClusterConfig(cluster-level, symmetric across all nodes) rather than per-node overrides.The comments in
kubernetes/infrastructure/post-crds/cilium-bgp.yaml(line 31–32) note "Each node sources from its eth0 (10.2.40.x / fd40::) via the kernel routing decision; no explicit localAddress override needed." This suggests that for the current two-peer design, switching to per-node configuration with explicitlocalAddressoverrides may be more complex than the proposal indicates—it could require migrating the entire BGP control-plane from cluster-wide to per-node configuration.Recommendation: Clarify whether step 4 (separating cluster-external) requires switching from
CiliumBGPClusterConfigtoCiliumBGPNodeConfigOverride, or whether the ingress peer can instead be added to the existing cluster-wide config with a newlocalAddressfield. The sandbox test should verify which approach works and whether they have different failure modes.2. IPv6 considerations not addressed (line 122–134)
The proposal focuses on IPv4, but the configuration already has native routing CIDRs defined for IPv6 (
ipv6-native-routing-cidr: "fd00::/8"). The migration steps should acknowledge whether the same traffic class separation applies to IPv6 peers and whether step 5 (native routing mode) requires addressing both families together or separately.3. Missing details on
directRoutingDevicebehavior testing (line 120–142)The open question is critical and correctly identified. The sandbox test should record:
cilium-dbg ip route listoutputs for direct routes (to see if next-hop changes)This will help avoid surprises during step 5 if the answer is unfavorable.
4. Step 2 (new VLANs on switch) is not actually "free" (line 149–150)
The proposal claims "Steps 2 and 3 are free." However, step 2 requires switch reconfiguration, which has lead time, vendor dependencies, and a non-zero risk of L2 misconfiguration (e.g., wrong port trunks, wrong VLAN IDs). While reversible, it is not risk-free and deserves acknowledgment that it depends on access to switch configuration and validation procedures.
🟢 Nits
Tests
This is a proposal document, not testable code, so traditional testing does not apply. However, the proposal correctly prioritizes a sandbox test as step 1. Before production rollout, verify:
nodePort.directRoutingDevicebehavior as specified in the open questionRecommendation
Looks reasonable — no blocking issues; suggestions above are optional clarifications. The proposal is honest about what is and is not known, prioritizes testing over production deployment, and follows a reversible migration strategy. Proceed with the sandbox phase as described in step 1.
Nachgetragen — die offene Frage war keine, sie stand im Quellcode.
installDirectRoutebekommt den Next-Hop als Parameter,nodeUpdateliefertGetNodeIP(false), und das bevorzugtNodeInternalIP. Der Wert kommt aus dem Kubernetes-Node-Objekt;--ipv4-nodesieht nach Override aus, aberinitFromK8s()läuft nachinitFromConfig()und überschreibt ihn wieder.--direct-routing-devicewählt ein Device, keine Adresse.Damit ist cluster-intern an die Kubelet-InternalIP geschweißt — und der Routing-Modus ist egal, weil Tunnel-Modus die VXLAN-Quelle aus derselben Variable nimmt. Native Routing bringt für dieses Problem nichts. Der einzige nicht-rollende Schritt aus dem ersten Entwurf fällt ersatzlos weg.
Also andersherum getrennt: nicht cluster-intern von der Node-IP wegziehen, sondern die Management-Dienste. Die teilen sich keinen Mechanismus, jeder einzeln geprüft:
cluster.etcd.advertisedSubnetsLayer2VIPConfig.linkapidantwortet auf jeder Node-Adresse (gemessen)Vier VLANs sind erreichbar, aber die Grenze läuft nicht dort, wo die Klassennamen sie vermuten lassen: kubelet bleibt bei cluster-intern. Das steht so drin, statt es unter Management einzusortieren und zu hoffen.
Empfehlung: Node-IP liegen lassen und VLAN 40 zum Cluster-VLAN umetikettieren, statt jedem Node eine neue Identität zu geben. Gleiches Ergebnis, und die Adresse, über die administriert wird, wechselt nie.
Nebenbei aufgefallen: k8s-1 hat seit der letzten Sitzung eine andere InternalIP (
.171→.169), der Cluster ist gesund. Deshalb nennt das Dokument keine Adresse als fix.🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commit55ea98c035095ec8c87fe24e7032db045daed074.Summary
This PR introduces a detailed technical proposal for separating cluster network traffic into four distinct VLANs: Ceph replication, cluster-internal, cluster-external, and management. The document provides a thorough analysis of Cilium/Talos architecture, identifies which traffic classes can be separated using existing configuration levers, and proposes a rolling, reversible migration path. The key insight—that cluster-internal traffic is bound to the node IP regardless of routing mode—reshapes the proposal into a more practical scope than originally framed in issue #576.
🟡 Suggestions
Line 62: The claim "In a Kubernetes cluster there is no path by which Cilium's node IP differs from the kubelet's
InternalIP" is strong. Slightly more precise to say "with this particular Cilium configuration" or "in this cluster's setup", since downstream Cilium versions or different init orderings could theoretically diverge.Line 90: "That is the honest result" is slightly informal for the technical tone of the document. Consider "This is the accurate result" or simply end the previous sentence with the technical finding.
Line 154: Step 6 says "Verify VLAN 40 by what is absent from it, not by what still works" — worth a sentence explaining what success looks like here (e.g., "confirm no management traffic appears on VLAN 40" or similar concrete metric).
Validation approach: The PR body (German text) mentions the next step is a sandbox test with two Talos VMs. Consider adding a brief "Next steps" section to the proposal itself stating this, so readers know the proposal is draft pending validation.
🟢 Nits
GetNodeIPprecedence into a separate bullet or sub-paragraph for clarity.apidanswered on a node address — consider hyperlink or fuller context on where this was measured (e.g., which node, command run).Tests
This is a documentation proposal, not code. No tests needed. The approach of validating in a sandbox environment (two Talos VMs) before actual recabling is sensible given the disruptive nature of network changes.
Recommendation
Looks reasonable — no blocking issues; comments above are optional refinements to clarity and completeness.