Share
Embed
LINE's next-generation SDN architecture
Toshiki Tsuchiya
LINE Service Network Team Infra Engineer
https://linedevday.linecorp.com/jp/2019/sessions/E1-2
LINE DevDay 2019
November 20, 2019
Transcript
45 slides
-
2019 DevDay LINE's Next-Generation SDN Architecture > Toshiki Tsuchiya >
LINE Service Network Team Infra Engineer -
LINE Services and Infrastructure Production Infrastructure Development Infrastructure Exclusive Infrastructure
Common Services (Messenger, Family Service, …) Fintech Services Exclusive Infrastructure -
LINE’s Infrastructure >Many works to design and build infrastructure >Lack
of infrastructure flexibility >Many fragmented infrastructure Challenges -
Solution: Multi-Tenant Network Underlay Network Overlay Network for Service C
Overlay Network for Service B Overlay Network for Service A Overlay Network > Service Specific Virtual Network > Flexible Network Policy > Simple & Scalable Physical Network Underlay Network -
Underlay Network CLOS Network > High-Capacity Network > Horizontally Scalable
Architecture Full L3 Network > All Nodes Are Connected With BGP • Simple, Stateless Network • Reduce Operation Cost … Spine Leaf ToR Server BGP -
> Less users, informations > Less device support > IP
CLOS awareness > Flexible instruction Pros Cons > Lose advance of Full-L3 underlay > Additional protocols for more flexible networking > More users, informations > Wider devices support Pros Cons How To Build Overlay Network? L2 Base Technology: VXLAN L3 Base Technology: SRv6 Adopted SRv6 -
Segment Routing > Source Routing Technology > Segment: Network Device,
Interface … • Segment ID (SID): IPv6 Address = 128bit SRv6 (IPv6 Segment Routing) A B D I/F I/F I/F I/F Segment -
SRv6 (IPv6 Segment Routing) A B C D X Y
SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet -
SRv6 (IPv6 Segment Routing) A B C D Insert Segment
List (Encap) X Y Packet SRH B::1,C::1,D::1 IPv6 H DA=B::1 SID: A::1 SID: B::1 SID: C::1 SID: D::1 -
SRv6 (IPv6 Segment Routing) A B C D X Y
SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet SRH B::1,C::1,D::1 IPv6 H DA=C::1 -
SRv6 (IPv6 Segment Routing) A B C D X Y
SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet SRH B::1,C::1,D::1 IPv6 H DA=D::1 -
SRv6 (IPv6 Segment Routing) A B C D X Y
SID: A::1 SID: B::1 SID: C::1 SID: D::1 Remove Segment List (Decap) Packet SRH B::1,C::1,D::1 IPv6 H DA=D::1 -
Multi-Tenant Network Underlay Network Tenant B Tenant A VM VM
VM VM Control-Plane > Data-Plane: Packet Processing > Control-Plane: Manage Tenants & Configure SRv6 Rules -
Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall,
IDS, …) CLOS Network Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, … -
Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall,
IDS, …) CLOS Network Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, … Linux Server Linux Server Linux Server Linux Server -
Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall,
IDS, …) Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, … SID = NN::A SID = HV1::A SID = NN::B SID = HV2::B NN::/96 HV1::/96 HV2::/96 NN::/96 -
Packet Flow in a Tenant NFV (Firewall, IDS, …) SRv6
Node Tenant Network Node A B Network Node A B Hypervisor1 (HV) A B VM1 VM2 Hypervisor2 (HV) A B VM3 VM4 IPv4 IPv6 IPv4 To VM3 HV2::A Encap Decap To VM3 To VM4 -
Packet Flow Between Tenants NFV (Firewall, IDS, …) SRv6 Node
Tenant Network Node A B Network Node A B Hypervisor1 (HV) A B VM1 VM2 Hypervisor2 (HV) A B VM3 VM4 IPv4 IPv6 IPv4 To VM4 NN::A Encap To VM4 HV2::B To VM3 To VM3 To VM4 Decap Decap Encap To VM4 -
> No Information on SRv6 Network Architecture > Linux Networking
Difficulties in Data-Plane Construction -
> Manage Tenants on Network Node & Hypervisor > Configure
Encap/Decap Rule to Network Node & Hypervisor Control-Plane Control-Plane NFV (Firewall, IDS, …) CLOS Network Network Node A B Network Node A B Hypervisor A B VM VM Hypervisor A B VM VM -
SRv6 Control-Plane Choices >ISIS >OSPF >BGP >SDN Controller LINE uses
OpenStack as Private Cloud Controller so adopted SDN Controller -
Neutron SRv6 Plugin - networking-sr > ML2 Mechanism/Type Driver and
Agent > Gateway Agent on Network Nodes > Service Plugin for New API To Add SRv6 Encap Rule Controller (Neutron) Service Plugin srv6_encap_network Type Driver srv6 Mechanism Driver mech_sr Network Node Hypervisor srgw_agent ml2 agent sr-agent -
Nova, Neutron Behavior - VM Create Neutron Controller Hypervisor Nova
nova-compute neutron-agent VM TAP 3. VM Info 4. Run VM 5. Create TAP 1. Create Network 1. Create Network 2. Create VM 2. Create VM 3. VM Info 4. Run VM 5. Create TAP -
Nova, Neutron Behavior - VM Create Neutron Controller Hypervisor Nova
nova-compute neutron-agent VM TAP 7. Get/Update Port Info VRF 6. Detect Tap 6. Detect Tap 7. Get/Update Port Info 8. Config Tap 9. Create VRF 10. Set SRv6 Encap/Decap Rules 8. Config Tap 9. Create VRF 10. Set SRv6 Encap/Decap Rules -
Packets for VM Encap/Decap on VRF Neutron Controller Hypervisor Nova
nova-compute neutron-agent VM TAP VRF IPv4 SRv6 IPv4 -
Set Encap Rule of Each VM Hypervisor 3 nova-compute neutron-agent
Hypervisor 1 neutron-agent VRF 1 VM1 VM2 VRF 1 VM5 TAP Set SRv6 Encap/Decap Rule Encap: VM1, VM2 → VRF1 of Hypervisor1 Encap: VM3, VM4 → VRF1 of Hypervisor2 Encap: VM5 → VRF1 of Hypervisor3 Hypervisor 2 neutron-agent VRF 1 VM3 VM4 Encap: VM5 → VRF1 of Hypervisor3 VM1 VM2 VM3 VM4 -
VM Configuration Summary > Communication Between VMs in the Same
Tenant Is Possible > Next: Communication Between VM and Other Networks Hypervisor VRF 1 VM VM Hypervisor VRF 1 VM VM VRF 2 Hypervisor VRF 2 VM VM Network Node VRF 1 VRF 2 Network Node VRF 1 VRF 2 -
Network Node Requirements: Multi Clusters Network Node VRF 1 VRF
2 VRF 3 Network Node VRF 1 VRF 2 VRF 3 OpenStack Cluster 1 OpenStack Cluster 2 OpenStack Cluster N ɾɾɾ -
Network Node Requirements: Scale Hypervisor VRF 1 VM VM Network
Node VRF 1 VRF 2 VRF 3 Network Node VRF 1 VRF 2 VRF 3 Hypervisor VRF 2 VM VM Hypervisor VRF 3 VM VM ɾɾɾ Network Node VRF 1 VRF 2 VRF 3 -
Etcd + Agent Model Network Node VRF 1 VRF 2
VRF 3 OpenStack Cluster 1 OpenStack Cluster 2 OpenStack Cluster N ɾɾɾ etcd Agent Network Node VRF 1 VRF 2 VRF 3 Agent -
Notify New Encap/Decap Rule via Etcd Network Node VRF Agent
etcd Neutron Controller Hypervisor Nova nova-compute neutron-agent VM TAP VRF 3. Put Port Info 4. Create VRF and Set SRv6 Encap/Decap Rules 2. Get/Update Port Info 1. Detect Tap 1. Detect Tap 2. Get/Update Port Info 3. Put Port Info 4. Create VRF and Set SRv6 Encap/Decap Rules -
Configuration Summary Hypervisor VRF 1 VM VM Hypervisor VRF 1
VM VM VRF 2 Hypervisor VRF 2 VM VM Network Node VRF 1 VRF 2 Network Node VRF 1 VRF 2 > Communication Between VMs in the Same Tenant Is Possible > Communication Between VM and Other Networks Is Possible -
> Follow the Design and Philosophy of OpenStack > Keep
It Simple Without Complicated Logic > Loose Coupling of Data-Plane and Control-Plane Control-Plane Design Policy -
> Multi-Tenant Network with SRv6 > Already deployed to the
Production Conclusion -
> Architecture Improvement • Service Chaining Future Plan NFV (Firewall,
IDS, …) Network Node (NN) A B Hypervisor2 (HV) A B VM3 VM4 VM Hypervisor1 (HV) A B VM2 VM1 NFV (Function Pool) Firewall (VM) IDS (VM) … IPv4 IPv6 IPv4 -
> Architecture Improvement • Service Chaining Future Plan Network Node
(NN) A B Hypervisor2 (HV) A B VM3 VM4 VM Hypervisor1 (HV) A B VM2 VM1 NFV (Function Pool) IPv4 IPv6 IPv4 Packet [to VM4] SRH NFV::FW, HV4::B Firewall (VM) IDS (VM) …