Save 37% off PRO during our Black Friday Sale! »

Designing and Implementing Multi-tenancy Data Center Networking with SRv6 in Large Scale Platform

Designing and Implementing Multi-tenancy Data Center Networking with SRv6 in Large Scale Platform

GTC 21
https://www.nvidia.com/ja-jp/gtc/

2021/4/14

Designing/Implementing Multi-tenancy Data Center Networking with SRv6 in Large Scale Platform
市原裕史 (Verda室 ネットワーク開発チーム)

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers
PRO

April 14, 2021
Tweet

Transcript

  1. Designing/Implementing Multi-tenancy Data Center Networking with SRv6 in Large Scale

    Platform Hirofumi Ichihara LINE corporation
  2. About Me • Hirofumi Ichihara • LINE Corporation ◦ Network

    Development Team • Network Software Developer ◦ SDN/NFV ◦ OpenStack Neutron ◦ Docker ◦ Kubernetes
  3. Private Cloud in LINE 3

  4. Verda & LINE Infra Scale 4 Virtual Machine 55000+ Baremetal

    server 20000+ Hypervisor 2000+ All Physical Servers 50000+ Peak of User Traffic 3Tbps+
  5. 5 FaaS IaaS PaaS KaaS Container Event Stream DBaaS DB

    Search and Analytics VM Identity Network Image DNS Block Storage Object Storage Bare metal Load Balancer Function
  6. LINE Services and Networks Full L3 CLOS Network* • Single

    tenant network • LINE message service and related services running Exclusive Network for Services • Service with specific requirements running • Building specific network for each service * Excitingly simple multi-path OpenStack networking: LAG-less, L2-less, yet fully redundant https://www.slideshare.net/linecorp/excitingly-simple-multipath-openstack-networking-lagless- l2less-yet-fully-redundant Many fragment underlay networks Many works to design and build Management cost increases Messaging, Manga, Game, ... Financial, HealthCare ...
  7. Multi tenant network overlay • Sharing underlay network decrease management

    cost • Achieve policy for each service(tenant) on isolated network Simple L3 underlay network Flexibly scale overlay network Security individual tenant Network Service Chaining
  8. 8 Virtual Private Network for Virtual Machine Private Network A

    VM VM Packet VM Private Network B VM VM VM • Virtual machine connects to private network • Isolated from other private networks • Multiple networks for each tenant
  9. VXLAN Pros • More information • Many network devices support

    Cons • Lose advances of full-L3 • Need additional protocol to achieve service chaining IPv6 Segment Routing (SRv6) Pros • IPv6 forwarding only on underlay • Support segregation and service chaining with Segment ID Cons • No information about DC use case • No network device support + SRv6 future Adopted SRv6 Multi-tenancy Network
  10. SRv6 Data Center Network Data Plane

  11. SRv6 Segment ID (SID) • 128bit number(IPv6 address) • Locator:

    Information for routing to SRv6 node(parent node). It must be unique whitin a SR domain • Function: Information to identify the action to be performed on the parent node Segment Routing Header (SRH) • IPv6 extension header • Including a Segment List, Segment Left points out current point of Segment List and so on Locator Function Function examples • H.Encaps(Encap): Encapsulation packet with IPv6 header and SRH • End.DX4(Decap): Remove IPv6 header and SRH from packet and then forward next hop • End.DT4(Decap): Remove IPv6 header and SRH from packet and then lookup routing table and forward (DT4 is implemented in Linux Kernel 5.11 but we donʼt support the kernel so we uses DX4 although DT4 is better) 128bit
  12. IPv6 Hypervisor Server 12 Networking Implementation with SRv6 VM VM

    VM VRF VRF Hypervisor Server VM VM VM VRF VRF SRv6 Header Packet • On Hypervisor server(HV), virtual network is achieved with Linux VRF • On IPv6 Network between HVs, virtual network is achieved with SRv6 overlay network
  13. Hypervisor Server A 13 H.Encaps VM VM vrfA Hypervisor Server

    B VM vrfA SRv6 Header fc00:aaaa:bbbb:cccd::2 Packet fc00:aaaa:bbbb:cccc::1/128 fc00:aaaa:bbbb:cccd::1/128 lo lo fc00:aaaa:bbbb:cccc::2/128 vrfA fc00:aaaa:bbbb:cccd::2/128 vrfA Locator Function Packet • Hypervisor server has specific SID identifies a VRF of server • Locator: Identify hypervisor server (e.g. fc00:aaaa:bbbb:cccc::/64) • Function: Identify VRF(e.g. 2) • SRv6 Header with SID is added to head of packet from VM
  14. Server VM 14 SRv6 Routing on CLOS Network • Our

    all L3 switches(Cumulus) donʼt support SRv6 • All switches forwards SRv6 packet as just IPv6 packet Server ToR ToR ToR ToR ToR ToR Server Server Server Server Server Server Server Server Server Server IPv6 Packet IPv6 Packet IPv6 Packet VM
  15. Hypervisor Server 15 End.DT4 VM VM vrfA Hypervisor Server VM

    vrfA SRv6 Header fc00:aaaa:bbbb:cccc::2 Packet fc00:aaaa:bbbb:cccc::1/128 lo fc00:aaaa:bbbb:cccc::2/128 vrfA Locator Function Packet • Hypervisor server has SRv6 decapsulation rule • The rule checks function in SRv6 Header and forwards the packet without the header • Packet is forwarded to VM on Linux VRF fc00:aaaa:bbbb:cccd::2/128 action End.DT4 vrfA Packet
  16. SRv6 Data Center Network Control Plane

  17. 17 LINE Network Control Plane Journey Full L3 Network with

    BGP Past 2016~ SRv6 Network with SDN Current 2019~ SRv6 Network with SDN and BGP Future 2021~
  18. 18 LINE Network Control Plane Journey Full L3 Network with

    BGP Full L3 Network with BGP Past 2016~ SRv6 Network with SDN Current 2019~ Future 2021~ SRv6 Network with SDN and BGP
  19. 19 Full L3 Network with BGP • OpenStack Nova creates

    Virtual Machine(VM) with tap device • Each tap device behaves like default gateway for each VM Nova Compute However there is no route to each VM VM Linux Routing Cumulus Switch
  20. 20 Full L3 Network with BGP • OpenStack Neutron adds

    a route for each VM on Hypervisor server Neutron Agent Now VMs get reachability to each VM but they donʼt have default route
  21. 21 Full L3 Network with BGP • L3 switch advertises

    default route to FRR on Hypervisor server • FRR gets default route and adds the route on Hypervisor server Now VMs get default route but VMʼs IP address is not reachable out side of Hypervisor server yet
  22. 22 Full L3 Network with BGP • FRR advertises VMʼs

    IP address to L3 switch • L3 switch gets VMʼs IP route and advertises it to upper L3 switches
  23. 23 Full L3 Network with BGP VMs get network reachability

    on CLOS Network
  24. 24 LINE Network Control Plane Journey SRv6 Network with SDN

    Full L3 Network with BGP Past 2016~ SRv6 Network with SDN Current 2019~ Future 2021~ SRv6 Network with SDN and BGP
  25. 25 SRv6 Network with SDN • Each VM connects to

    VRF • Each HV has IPv6 including SID Locator
  26. 26 SRv6 Network with SDN • FRR advertises IPv6 address

    route for each HV server • Neutron adds a route for each VM on VRF Packet with SRv6 header is reachable to each HV but VMʼs IP isnʼt reachable yet 73'#
  27. 27 SRv6 Network with SDN • Neutron generates SID for

    each VRF which combines Locator and Function • Neutron adds End.DT4 rules Packet with SRv6 header can be decapsulated on each HV server but VMʼs packet isnʼt encapsulated yet 73'#
  28. 28 SRv6 Network with SDN • Neutron adds H.Encaps rule

    to each VRF VMs get network reachability on SRv6 Network #
  29. 29 SRv6 Network with SDN Pain Point • All network

    configurations are managed via Neutron API • Failure of controller layer may affect network management • We have to add controller algorithm for everything • Sometimes API response latency may be bottleneck • Difficult to connect to SRv6 support network device • Neutron needs to configure network device if necessary
  30. 30 LINE Network Control Plane Journey SRv6 Network with SDN

    and BGP Full L3 Network with BGP Past 2016~ SRv6 Network with SDN Current 2019~ Future 2021~ SRv6 Network with SDN and BGP
  31. 31 SRv6 Network with SDN and BGP • Each VM

    connects to VRF • Each HV has IPv6 including SID Locator
  32. 32 SRv6 Network with SDN and BGP • FRR advertises

    IPv6 address route of each HV server • Neutron adds a route for each VM on VRF Packet with SRv6 header is reachable to each HV but VMʼs IP isnʼt reachable yet
  33. 33 SRv6 Network with SDN and BGP • Neutron generates

    SID for each VRF which combines Locator and Function • Neutron adds End.DT4 rules Packet with SRv6 header can be decapsulated on each HV server but VMʼs packet isnʼt encapsulated yet
  34. 34 SRv6 Network with SDN and BGP • FRR advertises

    VMʼs IP address and the SID information as VPNv4-Unicast via BGP
  35. 35 SRv6 Network with SDN and BGP • Upper L3

    switch(Cumulus) catches the advertised SRv6 route information
  36. 36 SRv6 Network with SDN and BGP • Upper L3

    switch(Cumulus) re-distributes VMʼs IP address and SRv6 information to FRR on each hypervisor server
  37. 37 SRv6 Network with SDN and BGP • FRR receives

    VMʼs IP address and SRv6 information from upper switch(Cumulus) • The route is installed on each server
  38. 38 FRR Contributions Under Review https://github.com/FRRouting/frr/pull/5865

  39. SRv6 Data Center Network NFV

  40. 40 NFV(Network Functions Virtualization) Achieve network function without exclusive physical

    device https://linedevday.linecorp.com/jp/2019/sessions/F1-7 https://linedevday.linecorp.com/2020/ja/sessions/2076
  41. 41 BGP to VM Enable VM to advertise routes via

    BGP Server L3 Switch L3 L3 Server Server Server Server Server VM 10.0.0.1 10.0.0.2 Server L3 Switch L3 L3 Server Server Server VM 10.0.0.1 10.0.0.2 10.0.1.1 10.0.1.2 10.0.0.1 10.0.0.2 10.0.1.1 10.0.1.2 10.0.0.1 10.0.0.2 BGP to VM Without BGP to VM
  42. 42 BGP to VM 1. HVʼs FRR advertises VMʼs IP

    address via BGP 2. Upper L3 switch(Cumulus) catches the advertised routes information
  43. 43 BGP to VM 1. HVʼs FRR advertises VMʼs IP

    address via BGP 2. Upper L3 switch(Cumulus) catches the advertised routes information 3. HVʼs FRR creates BGP peer to FRR on VM by Neutron • Use local AS number • Use metadata server IP(169.254.169.254) on HV
  44. 44 BGP to VM 1. HVʼs FRR advertises VMʼs IP

    address via BGP 2. Upper L3 switch(Cumulus) catches the advertised routes information 3. HVʼs FRR creates BGP peer to FRR on VM • Use local AS number • Use metadata server IP(169.254.169.254) on HV 4. Set advertised network address on VM
  45. 45 BGP to VM 1. HVʼs FRR advertises VMʼs IP

    address via BGP 2. Upper L3 switch(Cumulus) catches the advertised routes information 3. HVʼs FRR creates BGP peer to FRR on VM • Use local AS number • Use metadata server IP(169.254.169.254) on HV 4. Set advertised network address on VM 5. VMʼs FRR advertises the address to HVʼs FRR via BGP
  46. 46 BGP to VM 1. HVʼs FRR advertises VMʼs IP

    address via BGP 2. Upper L3 switch(Cumulus) catches the advertised routes information 3. HVʼs FRR creates BGP peer to FRR on VM • Use local AS number • Use metadata server IP(169.254.169.254) on HV 4. Set advertised network address on VM 5. VMʼs FRR advertises the address to HVʼs FRR via BGP 6. HVʼs FRR advertises the address to upper L3 switch
  47. 47 Cloud Router IPv6 Hypervisor Server VM VM VM Hypervisor

    Server Cloud Router Cloud Router SRv6 Header Packet • SRv6 network between VMs with same private network • IPv4 network between VM and IDC network via Cloud Router • Secure network between VM and other network via Cloud Router IPv4 Packet Another location VPN
  48. 48 Cloud Router • Cloud Router runs in VM •

    Cloud Router isnʼt share over a tenant • Multiple Cloud Routers can run in a tenant
  49. 49 Cloud Router NAT 1. Cloud Router advertises NAT addresses

  50. 50 Cloud Router NAT 1. Cloud Router advertises NAT addresses

    2. Cloud Router changes source address to NAT address and then forward packets
  51. 51 Cloud Router Routing 1. Cloud Router advertises Network3ʼs subnet

    address and then forward packets
  52. 52 Cloud Router VPN 1. Cloud Router creates IPsec tunnel

    to VPN Gateway on other location 2. Cloud Router advertises Network1ʼs subnet address to VPN Gateway 3. Could Router forwards packets into IPsec tunnel
  53. 53 Summary • Multi-tenancy data center networking with SRv6 use

    case • Architecture of SRv6 data networking • SRv6 SDN implementation • NFV implementation on SRv6 network • Future plan: SDN and BGP hybrid model Thank you