LINE's next-generation SDN architecture

Toshiki Tsuchiya
LINE Service Network Team Infra Engineer
https://linedevday.linecorp.com/jp/2019/sessions/E1-2

Transcript

1. 2019 DevDay LINE's Next-Generation SDN Architecture > Toshiki Tsuchiya > LINE Service Network Team Infra Engineer
2. LINE’s Infrastructure
3. LINE Services and Infrastructure On-Premises Infrastructure …
4. LINE Services and Infrastructure Production Infrastructure Development Infrastructure Exclusive Infrastructure Common Services 
 (Messenger, Family Service, …) Fintech Services Exclusive Infrastructure
5. LINE’s Infrastructure >Many works to design and build infrastructure >Lack of infrastructure flexibility >Many fragmented infrastructure Challenges
6. Solution: Multi-Tenant Network Underlay Network Overlay Network for Service C Overlay Network for Service B Overlay Network for Service A Overlay Network > Service Specific Virtual Network > Flexible Network Policy > Simple & Scalable Physical Network Underlay Network
7. Multi-Tenant Network
8. Underlay Network CLOS Network > High-Capacity Network > Horizontally Scalable Architecture Full L3 Network > All Nodes Are Connected With BGP • Simple, Stateless Network • Reduce Operation Cost … Spine Leaf ToR Server BGP
9. > Less users, informations > Less device support > IP CLOS awareness > Flexible instruction Pros Cons > Lose advance of Full-L3 underlay > Additional protocols for
 more flexible networking > More users, informations > Wider devices support Pros Cons How To Build Overlay Network? L2 Base Technology: VXLAN L3 Base Technology: SRv6 Adopted SRv6
10. Segment Routing > Source Routing Technology > Segment: Network Device, Interface … • Segment ID (SID): IPv6 Address = 128bit SRv6 (IPv6 Segment Routing) A B D I/F I/F I/F I/F Segment
11. SRv6 (IPv6 Segment Routing) A B C D X Y SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet
12. SRv6 (IPv6 Segment Routing) A B C D Insert Segment List (Encap) X Y Packet SRH B::1,C::1,D::1 IPv6 H DA=B::1 SID: A::1 SID: B::1 SID: C::1 SID: D::1
13. SRv6 (IPv6 Segment Routing) A B C D X Y SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet SRH B::1,C::1,D::1 IPv6 H DA=C::1
14. SRv6 (IPv6 Segment Routing) A B C D X Y SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet SRH B::1,C::1,D::1 IPv6 H DA=D::1
15. SRv6 (IPv6 Segment Routing) A B C D X Y SID: A::1 SID: B::1 SID: C::1 SID: D::1 Remove Segment List (Decap) Packet SRH B::1,C::1,D::1 IPv6 H DA=D::1
16. Multi-Tenant Network Underlay Network Tenant B Tenant A VM VM VM VM Control-Plane > Data-Plane: Packet Processing > Control-Plane: Manage Tenants & Configure SRv6 Rules
17. Multi-Tenant Network
 Data-Plane
18. Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall, IDS, …) CLOS Network Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, …
19. Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall, IDS, …) CLOS Network Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, … Linux Server Linux Server Linux Server Linux Server
20. Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall, IDS, …) Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, … SID = NN::A SID = HV1::A SID = NN::B SID = HV2::B NN::/96 HV1::/96 HV2::/96 NN::/96
21. Packet Flow in a Tenant NFV (Firewall, IDS, …) SRv6 Node Tenant Network Node A B Network Node A B Hypervisor1 (HV) A B VM1 VM2 Hypervisor2 (HV) A B VM3 VM4 IPv4 IPv6 IPv4 To VM3 HV2::A Encap Decap To VM3 To VM4
22. Packet Flow Between Tenants NFV (Firewall, IDS, …) SRv6 Node Tenant Network Node A B Network Node A B Hypervisor1 (HV) A B VM1 VM2 Hypervisor2 (HV) A B VM3 VM4 IPv4 IPv6 IPv4 To VM4 NN::A Encap To VM4 HV2::B To VM3 To VM3 To VM4 Decap Decap Encap To VM4
23. > No Information on SRv6 Network Architecture > Linux Networking Difficulties in Data-Plane Construction
24. Multi-Tenant Network
 Control-Plane
25. > Manage Tenants on Network Node & Hypervisor > Configure Encap/Decap Rule to Network Node & Hypervisor Control-Plane Control-Plane NFV (Firewall, IDS, …) CLOS Network Network Node A B Network Node A B Hypervisor A B VM VM Hypervisor A B VM VM
26. SRv6 Control-Plane Choices >ISIS >OSPF >BGP >SDN Controller LINE uses OpenStack as Private Cloud Controller so adopted SDN Controller
27. OpenStack >Cloud Operating System >Neutron: OpenStack Networking Controller • Include APIs and Plugins
28. Neutron SRv6 Plugin - networking-sr > ML2 Mechanism/Type Driver and Agent > Gateway Agent on Network Nodes > Service Plugin for New API To Add SRv6 Encap Rule Controller (Neutron) Service Plugin srv6_encap_network Type Driver srv6 Mechanism Driver mech_sr Network Node Hypervisor srgw_agent ml2 agent sr-agent
29. Nova, Neutron Behavior - VM Create Neutron Controller Hypervisor Nova nova-compute neutron-agent VM TAP 3. VM Info 4. Run VM 5. Create TAP 1. Create Network 1. Create Network 2. Create VM 2. Create VM 3. VM Info 4. Run VM 5. Create TAP
30. Nova, Neutron Behavior - VM Create Neutron Controller Hypervisor Nova nova-compute neutron-agent VM TAP 7. Get/Update Port Info VRF 6. Detect Tap 6. Detect Tap 7. Get/Update Port Info 8. Config Tap 9. Create VRF 10. Set SRv6 Encap/Decap Rules 8. Config Tap 9. Create VRF 10. Set SRv6 Encap/Decap Rules
31. Packets for VM Encap/Decap on VRF Neutron Controller Hypervisor Nova nova-compute neutron-agent VM TAP VRF IPv4 SRv6 IPv4
32. Set Encap Rule of Each VM Hypervisor 3 nova-compute neutron-agent Hypervisor 1 neutron-agent VRF 1 VM1 VM2 VRF 1 VM5 TAP Set SRv6 Encap/Decap Rule Encap: VM1, VM2 → VRF1 of Hypervisor1 Encap: VM3, VM4 → VRF1 of Hypervisor2 Encap: VM5 → VRF1 of Hypervisor3 Hypervisor 2 neutron-agent VRF 1 VM3 VM4 Encap: VM5 → VRF1 of Hypervisor3 VM1 VM2 VM3 VM4
33. VM Configuration Summary > Communication Between VMs in the Same Tenant Is Possible > Next: Communication Between VM and Other Networks Hypervisor VRF 1 VM VM Hypervisor VRF 1 VM VM VRF 2 Hypervisor VRF 2 VM VM Network Node VRF 1 VRF 2 Network Node VRF 1 VRF 2
34. Network Node Requirements: Multi Clusters Network Node VRF 1 VRF 2 VRF 3 Network Node VRF 1 VRF 2 VRF 3 OpenStack Cluster 1 OpenStack Cluster 2 OpenStack Cluster N ɾɾɾ
35. Network Node Requirements: Scale Hypervisor VRF 1 VM VM Network Node VRF 1 VRF 2 VRF 3 Network Node VRF 1 VRF 2 VRF 3 Hypervisor VRF 2 VM VM Hypervisor VRF 3 VM VM ɾɾɾ Network Node VRF 1 VRF 2 VRF 3
36. Etcd + Agent Model Network Node VRF 1 VRF 2 VRF 3 OpenStack Cluster 1 OpenStack Cluster 2 OpenStack Cluster N ɾɾɾ etcd Agent Network Node VRF 1 VRF 2 VRF 3 Agent
37. Notify New Encap/Decap Rule via Etcd Network Node VRF Agent etcd Neutron Controller Hypervisor Nova nova-compute neutron-agent VM TAP VRF 3. Put Port Info 4. Create VRF and Set SRv6 Encap/Decap Rules 2. Get/Update Port Info 1. Detect Tap 1. Detect Tap 2. Get/Update Port Info 3. Put Port Info 4. Create VRF and Set SRv6 Encap/Decap Rules
38. Configuration Summary Hypervisor VRF 1 VM VM Hypervisor VRF 1 VM VM VRF 2 Hypervisor VRF 2 VM VM Network Node VRF 1 VRF 2 Network Node VRF 1 VRF 2 > Communication Between VMs in the Same Tenant Is Possible > Communication Between VM and Other Networks Is Possible
39. > Follow the Design and Philosophy of OpenStack > Keep It Simple Without Complicated Logic > Loose Coupling of Data-Plane and Control-Plane Control-Plane Design Policy
40. > Multi-Tenant Network with SRv6 > Already deployed to the Production Conclusion
41. > Performance Improvement • XDP, DPDK, NIC Offloading Future Plan
42. > Architecture Improvement • Service Chaining Future Plan NFV (Firewall, IDS, …) Network Node (NN) A B Hypervisor2 (HV) A B VM3 VM4 VM Hypervisor1 (HV) A B VM2 VM1 NFV (Function Pool) Firewall (VM) IDS (VM) … IPv4 IPv6 IPv4
43. > Architecture Improvement • Service Chaining Future Plan Network Node (NN) A B Hypervisor2 (HV) A B VM3 VM4 VM Hypervisor1 (HV) A B VM2 VM1 NFV (Function Pool) IPv4 IPv6 IPv4 Packet [to VM4] SRH NFV::FW, HV4::B Firewall (VM) IDS (VM) …
44. SRv6 Contribution Internet Draft draft-matsushima-spring-srv6-deployment-status > IETF106: 11/16 - 11/22
45. Thank You