Infrastructure Exclusive Infrastructure Common services (messenger, family service, …) Fintech services ... •Many fragmented infrastructure :( ◦ Many works to design and build network ◦ Lack of infrastructure flexibility On premises Infrastructure Integrated Infrastructure is described at here https://engineering.linecorp.com/en/blog/openstack-summit-vancouver-2018-recap-2-2/ https://www.janog.gr.jp/meeting/janog43/application/files/7915/4823/1858/janog43-line-kobayashi.pdf Exclusive Infrastructure
• complexity for L2 extension • need additional proto for SFC SRv6 Pros • flexible instructions with SID • IP-Fabric awareness Cons • less device support • less users and examples We’ve decided to adopt SRv6 for its innovative design. Multi-tenancy :: Which Technology
Node (SRv6 Node) Router Switch Switch Switch Switch Switch Switch Hypervisor (SRv6 Node) Hypervisor (SRv6 Node) Hypervisor (SRv6 Node) VM Tenant A VM Tenant B VM Tenant A VM Tenant B VM Tenant A VM Tenant B NFV (FW, IDS, ...) Transit Node IPv6 forwarding only without process for SRH Hypervisor (HV) • From VM → Encap • To VM → Decap Network Node (NN) • Legacy network/Internet/Tenants SRv6 Data Plane :: Architecture Overview Network Node (SRv6 Node) SRv6 unaware device
SID: C2::A VM A1 C1::/96 Network Node2 C1::/96 Network Node1 VRF Tenant A SID: C1::A Routing to SID :: Use eBGP • Create VRF(l3mdev) for each tenant on NN, HV • Assign IPv6 /96 block (Locator) to NN, HV ◦ Advertise /96 IPv6 address(Locator) via BGP (Network Node adv anycast-SID) ◦ Add identifier for each tenant to the Locator as Function (LINE uses specific address from 169.254.0.0/16 each tenant) • Use T.Encaps and End.DX4 only (We configure the End.DT4 as pseudo way) please see the appendix. :) • Importance: is ECMP available on your NW …? VRF Tenant B SID: C2::B VM B1 Hypervisor2 C3::/96 VRF Tenant A SID: C3::A VM A2 VRF Tenant B SID: C3::B VM B2 VRF Tenant B SID: C1::B VRF Tenant A SID: C1::A VRF Tenant B SID: C1::B Route Advertise (eBGP)
Router Hypervisor1 C2::/96 NFV VRF1 Tenant A SID: C2::A VM A1 C1::/96 Network Node2 C1::/96 Network Node1 VRF1 Tenant A SID: C1::A VRF2 Tenant B SID: C2::B VM B1 Hypervisor2 C3::/96 VRF1 Tenant A SID: C3::A VM A2 VRF2 Tenant B SID: C3::B VM B2 VRF2 Tenant B SID: C1::B VRF1 Tenant A SID: C1::A VRF2 Tenant B SID: C1::B T.Encaps dst = C3::A End.DX4 nexthop VRF1 VM A1 (HV1 Tennant A) → VM A2 (HV2 Tennant A)
SID: C2::A VM A1 C1::/96 Network Node2 C1::/96 Network Node1 VRF1 Tenant A SID: C1::A VRF2 Tenant B SID: C2::B VM B1 Hypervisor2 C3::/96 VRF1 Tenant A SID: C3::A VM A2 VRF2 Tenant B SID: C3::B VM B2 VRF2 Tenant B SID: C1::B VRF1 Tenant A SID: C1::A VRF2 Tenant B SID: C1::B Data Plane :: Different Tenants Packet flow VM A1 (HV1 Tennant A) → VM B2 (HV2 Tennant B) T.Encaps dst = C1::A End.DX4 nexthop VRF1 T.Encaps dst = C3::B End.DX4 nexthop VRF2
BGP, SDN-controller • Requirements: ◦ scalable (network node, hypervisor, multi-openstack-cluster) ◦ can be connected from another network • Design Principle: ◦ follow the design and philosophy of Openstack ◦ simple, Loose coupling D/C-plane LINE uses OpenStack as Private Cloud Controller so adopted SDN Controller
agent ◦ srv6: new network type driver ◦ mech_sr: new mechanism driver Controller node neutron server type driver srv6 mechanism driver mech_sr Service Plugin srv6_encap_network Hypervisor ml2 agent sr-agent Network node ml2 agent srgw-agent (2) 2 types of neutron agent ◦ SR-gw-agent on Network-Node ◦ SR-agent on Hypervisor (3) Service plugin for new API to add SRv6 encap rule
Hypervisor nova-compute 6. Detect tap VM tap neutron-agent 8. Config tap VRF 9. Create VRF 10. Set SRv6 encap/decap rules 7. Get/Update port Info 1. Create Network 2. Create VM 4. Run VM 5. Create tap 3. VM Info Nova Please check the appendix-3 about the information in the step7.
2 ・・・ cluste r 2 vrf 1 cluste r 3 vrf 1 OpenStack Cluster 1 OpenStack Cluster 2 OpenStack Cluster N ・・・ cluste r 1 vrf 1 cluste r 2 vrf 1 cluste r 3 vrf 1 cluste r 1 vrf 1 cluste r 2 vrf 1 cluste r 3 vrf 1 Network Node Requirements: Multi OpenStack Clusters and Scale Hypervisor VRF VM VM Hypervisor VRF VM VM Hypervisor VRF VM VM Network 1 Network N VRF Network 2 ・・・ VRF VRF VRF VRF VRF VRF VRF VRF
3. Put port info Network srgw-agent VRF 4. Get changes 5. Create VRF and Set SRv6 encap/decap rules Hypervisor nova-compute 1. Detect tap VM tap 2. Get/ Update port Info sr-agent VRF
◦ T.Encaps and End.DX4 only ◦ use the magic to provide pseudo End.DT4 using End.DX4 and vrf redirection ◦ underlay is typical clos-fabric (bgp-unnumbered and anycast) • Control plane ◦ develop networking-sr as neutron ML2 plugin ◦ scale (multi network-node, multi openstack-cluster) with etcd ◦ can be connected from another network with new-api • Next ◦ Performance improvement (see appendix-5) ◦ Control plane mechanizm improvement using BGP
;) eBGP C-Plane Neutron C-Plane Underlay Network Overlay Network for Service C Overlay Network for Service B Overlay Network for Service A ipv6 ucast sr agnet srgw agnet Underlay Network Overlay Network for Service C Overlay Network for Service B Overlay Network for Service A ipv6 ucast vpnv4 ucast lightweight agent • BGP Integration using SRv6-VPNv4 • with Yet-Another SRv6-VPNv4 Implementation …? • Currently we are working for this new-control-plane • CAUTION :: THIS IS JUST A PLAN
for Service C Overlay Network for Service B Overlay Network for Service A • 利点 ◦ Etcdを用いたスケーラブルな設計 ◦ Neutron側でnetwork tenant等をすべて掌握 ◦ 基本的に初期に検討された要件はクリア! • 欠点 ◦ Neutron側の過負荷 (VM作成時のcplaneトラフィック等) ◦ Network Nodeの仕事が多い (複数のOpenstack clusterの管理等) eBGP C-plane ipv6 ucast Neutron C-Plane sr agnet srgw agnet
bgpを用いたスケーラブルな設計 ◦ Neutron側の負担軽減 ◦ よりコモディティな設計思想 Network Nodeの仕事を単純化 • 欠点 ◦ 実装の少なさ (OSSは皆無) ◦ 前回同様, 取り組みの前例が少ない Underlay Network Overlay Network for Service C Overlay Network for Service B Overlay Network for Service A eBGP C-Plane Neutron C-Plane ipv6 ucast vpnv4 ucast lightweight agent
SRv6 DCN with Openstack :: Next Step • Data plane • Performance improvement (see appendix-5) • Control plane • BGP VPNv4 Integration • neutron :: Less Complication • frr :: standard MP-BGP SRv6 VPNv4 • We are working for VPNv4-SRv6 implementation :) • Current • End.DX4 & T.Encaps only • networking-sr :: Neutron ML2 plugin ここで更に議論しましょう!
vrf vrf1 10.0.0.113 encap seg6 mode encap segs 1 [ 2001::ffc1:108 ] dev vrf5c0594737b87 scope link 10.0.0.114 encap seg6 mode encap segs 1 [ 2001::ffc1:108 ] dev vrf5c0594737b87 scope link 10.0.0.115 encap seg6 mode encap segs 1 [ 2001::ffc2:108 ] dev vrf5c0594737b87 scope link Locator(HV Address) Function (ID of each tenant) Encap [NetworkNode]# ip -6 route show table local local 2001::aaaa:102 encap seg6local action End.DX4 nh4 169.254.1.2 dev vrf2 local 2001::aaaa:104 encap seg6local action End.DX4 nh4 169.254.1.4 dev vrf4 local 2001::aaaa:108 encap seg6local action End.DX4 nh4 169.254.1.8 dev vrf8 Decap Locator(NN Address) Function (Tenant identifier) IPv4 address to identify each tenant. They are assigned to VRF IF (That is magic to lookup VRF in End.DX4) Destination IPv4 address of VM SID List (sorry, address is dummy number)
nova-compute Nova neutron-agent VM Configuration 1. Create network 2. Create VM 3. Notify VM info 4. Run VM 5. Create tap NW Configuration 6. Detect tap 7. Update/Get port info 8. Config tap 9. Create VRF 10. Set SRv6 encap/decap rules 7. Get/Update port Info No.7 VRF info in port/binding:profile { "port":{ "binding:profile": { "segment_node_id": "2001::ffc1", # Locator where VM with the port running "vrf": "vrf1", # VRF IF name for the port. The name is combined by "vrf" + tenant_id + network_id "vrf_cidr": "169.254.1.0/24", # IP CIDR of VRF for the port "vrf_ip": "169.254.1.1" # IP Address of VRF for the port } } }
rule • {network,tenant,project}_id: ID for nw tenant • encap_rules: SRv6 encap rule list ◦ destination: addr for specified dest (VIP) ◦ nexthop: SID list for SRH • id: ID for encap rule srv6_encap_network resource
the auto-loadtest-environment. • SRv6 forwarding / IPv6 forwarding ◦ comparison between each software-dataplanes ◦ TSO evaluation ◦ performance impact by new-feature • automatically test and report (generate figures)
sub type 1: Label Index (SR-MPLS) ◦ sub type 2: IPv6 SID ◦ sub type 3: Originator SRGB (SR-MPLS) ◦ sub type 5: SRv6 L3 Service (new) ◦ sub type 6: SRv6 L2 Service (new) • BGP Prefix-SID Path-attr ◦ attribute type=40 (BGP-Prefix-SID) ◦ https://tools.ietf.org/html/draft-ietf-idr-bgp-prefix-sid-27 • WiresharkでPrefix-SIDのType-1,2,3は見れる ◦ Type-5(SRv6 L3 Service)のDissectorは別途開発 ◦ 今後upstreamに提案予定