LINE's next-generation SDN architecture

LINE's next-generation SDN architecture

Toshiki Tsuchiya
LINE Service Network Team Infra Engineer
https://linedevday.linecorp.com/jp/2019/sessions/E1-2

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay LINE's Next-Generation SDN Architecture > Toshiki Tsuchiya >

    LINE Service Network Team Infra Engineer
  2. LINE’s Infrastructure

  3. LINE Services and Infrastructure On-Premises Infrastructure …

  4. LINE Services and Infrastructure Production Infrastructure Development Infrastructure Exclusive Infrastructure

    Common Services 
 (Messenger, Family Service, …) Fintech Services Exclusive Infrastructure
  5. LINE’s Infrastructure >Many works to design and build infrastructure >Lack

    of infrastructure flexibility >Many fragmented infrastructure Challenges
  6. Solution: Multi-Tenant Network Underlay Network Overlay Network for Service C

    Overlay Network for Service B Overlay Network for Service A Overlay Network > Service Specific Virtual Network > Flexible Network Policy > Simple & Scalable Physical Network Underlay Network
  7. Multi-Tenant Network

  8. Underlay Network CLOS Network > High-Capacity Network > Horizontally Scalable

    Architecture Full L3 Network > All Nodes Are Connected With BGP • Simple, Stateless Network • Reduce Operation Cost … Spine Leaf ToR Server BGP
  9. > Less users, informations > Less device support > IP

    CLOS awareness > Flexible instruction Pros Cons > Lose advance of Full-L3 underlay > Additional protocols for
 more flexible networking > More users, informations > Wider devices support Pros Cons How To Build Overlay Network? L2 Base Technology: VXLAN L3 Base Technology: SRv6 Adopted SRv6
  10. Segment Routing > Source Routing Technology > Segment: Network Device,

    Interface … • Segment ID (SID): IPv6 Address = 128bit SRv6 (IPv6 Segment Routing) A B D I/F I/F I/F I/F Segment
  11. SRv6 (IPv6 Segment Routing) A B C D X Y

    SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet
  12. SRv6 (IPv6 Segment Routing) A B C D Insert Segment

    List (Encap) X Y Packet SRH B::1,C::1,D::1 IPv6 H DA=B::1 SID: A::1 SID: B::1 SID: C::1 SID: D::1
  13. SRv6 (IPv6 Segment Routing) A B C D X Y

    SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet SRH B::1,C::1,D::1 IPv6 H DA=C::1
  14. SRv6 (IPv6 Segment Routing) A B C D X Y

    SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet SRH B::1,C::1,D::1 IPv6 H DA=D::1
  15. SRv6 (IPv6 Segment Routing) A B C D X Y

    SID: A::1 SID: B::1 SID: C::1 SID: D::1 Remove Segment List (Decap) Packet SRH B::1,C::1,D::1 IPv6 H DA=D::1
  16. Multi-Tenant Network Underlay Network Tenant B Tenant A VM VM

    VM VM Control-Plane > Data-Plane: Packet Processing > Control-Plane: Manage Tenants & Configure SRv6 Rules
  17. Multi-Tenant Network
 Data-Plane

  18. Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall,

    IDS, …) CLOS Network Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, …
  19. Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall,

    IDS, …) CLOS Network Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, … Linux Server Linux Server Linux Server Linux Server
  20. Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall,

    IDS, …) Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, … SID = NN::A SID = HV1::A SID = NN::B SID = HV2::B NN::/96 HV1::/96 HV2::/96 NN::/96
  21. Packet Flow in a Tenant NFV (Firewall, IDS, …) SRv6

    Node Tenant Network Node A B Network Node A B Hypervisor1 (HV) A B VM1 VM2 Hypervisor2 (HV) A B VM3 VM4 IPv4 IPv6 IPv4 To VM3 HV2::A Encap Decap To VM3 To VM4
  22. Packet Flow Between Tenants NFV (Firewall, IDS, …) SRv6 Node

    Tenant Network Node A B Network Node A B Hypervisor1 (HV) A B VM1 VM2 Hypervisor2 (HV) A B VM3 VM4 IPv4 IPv6 IPv4 To VM4 NN::A Encap To VM4 HV2::B To VM3 To VM3 To VM4 Decap Decap Encap To VM4
  23. > No Information on SRv6 Network Architecture > Linux Networking

    Difficulties in Data-Plane Construction
  24. Multi-Tenant Network
 Control-Plane

  25. > Manage Tenants on Network Node & Hypervisor > Configure

    Encap/Decap Rule to Network Node & Hypervisor Control-Plane Control-Plane NFV (Firewall, IDS, …) CLOS Network Network Node A B Network Node A B Hypervisor A B VM VM Hypervisor A B VM VM
  26. SRv6 Control-Plane Choices >ISIS >OSPF >BGP >SDN Controller LINE uses

    OpenStack as Private Cloud Controller so adopted SDN Controller
  27. OpenStack >Cloud Operating System >Neutron: OpenStack Networking Controller • Include

    APIs and Plugins
  28. Neutron SRv6 Plugin - networking-sr > ML2 Mechanism/Type Driver and

    Agent > Gateway Agent on Network Nodes > Service Plugin for New API To Add SRv6 Encap Rule Controller (Neutron) Service Plugin srv6_encap_network Type Driver srv6 Mechanism Driver mech_sr Network Node Hypervisor srgw_agent ml2 agent sr-agent
  29. Nova, Neutron Behavior - VM Create Neutron Controller Hypervisor Nova

    nova-compute neutron-agent VM TAP 3. VM Info 4. Run VM 5. Create TAP 1. Create Network 1. Create Network 2. Create VM 2. Create VM 3. VM Info 4. Run VM 5. Create TAP
  30. Nova, Neutron Behavior - VM Create Neutron Controller Hypervisor Nova

    nova-compute neutron-agent VM TAP 7. Get/Update Port Info VRF 6. Detect Tap 6. Detect Tap 7. Get/Update Port Info 8. Config Tap 9. Create VRF 10. Set SRv6 Encap/Decap Rules 8. Config Tap 9. Create VRF 10. Set SRv6 Encap/Decap Rules
  31. Packets for VM Encap/Decap on VRF Neutron Controller Hypervisor Nova

    nova-compute neutron-agent VM TAP VRF IPv4 SRv6 IPv4
  32. Set Encap Rule of Each VM Hypervisor 3 nova-compute neutron-agent

    Hypervisor 1 neutron-agent VRF 1 VM1 VM2 VRF 1 VM5 TAP Set SRv6 Encap/Decap Rule Encap: VM1, VM2 → VRF1 of Hypervisor1 Encap: VM3, VM4 → VRF1 of Hypervisor2 Encap: VM5 → VRF1 of Hypervisor3 Hypervisor 2 neutron-agent VRF 1 VM3 VM4 Encap: VM5 → VRF1 of Hypervisor3 VM1 VM2 VM3 VM4
  33. VM Configuration Summary > Communication Between VMs in the Same

    Tenant Is Possible > Next: Communication Between VM and Other Networks Hypervisor VRF 1 VM VM Hypervisor VRF 1 VM VM VRF 2 Hypervisor VRF 2 VM VM Network Node VRF 1 VRF 2 Network Node VRF 1 VRF 2
  34. Network Node Requirements: Multi Clusters Network Node VRF 1 VRF

    2 VRF 3 Network Node VRF 1 VRF 2 VRF 3 OpenStack Cluster 1 OpenStack Cluster 2 OpenStack Cluster N ɾɾɾ
  35. Network Node Requirements: Scale Hypervisor VRF 1 VM VM Network

    Node VRF 1 VRF 2 VRF 3 Network Node VRF 1 VRF 2 VRF 3 Hypervisor VRF 2 VM VM Hypervisor VRF 3 VM VM ɾɾɾ Network Node VRF 1 VRF 2 VRF 3
  36. Etcd + Agent Model Network Node VRF 1 VRF 2

    VRF 3 OpenStack Cluster 1 OpenStack Cluster 2 OpenStack Cluster N ɾɾɾ etcd Agent Network Node VRF 1 VRF 2 VRF 3 Agent
  37. Notify New Encap/Decap Rule via Etcd Network Node VRF Agent

    etcd Neutron Controller Hypervisor Nova nova-compute neutron-agent VM TAP VRF 3. Put Port Info 4. Create VRF and Set SRv6 Encap/Decap Rules 2. Get/Update Port Info 1. Detect Tap 1. Detect Tap 2. Get/Update Port Info 3. Put Port Info 4. Create VRF and Set SRv6 Encap/Decap Rules
  38. Configuration Summary Hypervisor VRF 1 VM VM Hypervisor VRF 1

    VM VM VRF 2 Hypervisor VRF 2 VM VM Network Node VRF 1 VRF 2 Network Node VRF 1 VRF 2 > Communication Between VMs in the Same Tenant Is Possible > Communication Between VM and Other Networks Is Possible
  39. > Follow the Design and Philosophy of OpenStack > Keep

    It Simple Without Complicated Logic > Loose Coupling of Data-Plane and Control-Plane Control-Plane Design Policy
  40. > Multi-Tenant Network with SRv6 > Already deployed to the

    Production Conclusion
  41. > Performance Improvement • XDP, DPDK, NIC Offloading Future Plan

  42. > Architecture Improvement • Service Chaining Future Plan NFV (Firewall,

    IDS, …) Network Node (NN) A B Hypervisor2 (HV) A B VM3 VM4 VM Hypervisor1 (HV) A B VM2 VM1 NFV (Function Pool) Firewall (VM) IDS (VM) … IPv4 IPv6 IPv4
  43. > Architecture Improvement • Service Chaining Future Plan Network Node

    (NN) A B Hypervisor2 (HV) A B VM3 VM4 VM Hypervisor1 (HV) A B VM2 VM1 NFV (Function Pool) IPv4 IPv6 IPv4 Packet [to VM4] SRH NFV::FW, HV4::B Firewall (VM) IDS (VM) …
  44. SRv6 Contribution Internet Draft draft-matsushima-spring-srv6-deployment-status > IETF106: 11/16 - 11/22

  45. Thank You