What Happened in Data Center Networking?

Slide 1

Slide 1 text

What Happened in Data Center Networking? Phil Huang Open Networking Solution Engineer, Edgecore Networks Digital Ocean HsinChu, Taiwan, Aug 25, 2017

Slide 2

Slide 2 text

Phil Huang 黃秉鈞 (小飛機) • Edgecore Networks Solution Engineer • ONF CORD / Atrium • BigSwitch / Pica8 / CumulusLinux • Open Source SI • ONF CORD Ambassador • SDNDS-TW Co-Founder 8/26/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com 2

Slide 3

Slide 3 text

Why Edgecore Networks? 8/26/17 3 Delivering at Scale Cumulus® Linux® Network OS ICOS Software Network OS Partner SONiC Open Hardware / Open Source Software Delivery & Support

Slide 4

Slide 4 text

Network Evolution

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Underlay Network Evolution for Data Center © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Three-Tier Architecture Ref: https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/ Leaf-Spine Architecture 1 3 2 4 Facebook Fabric

Slide 8

Slide 8 text

Overlay Network Evolution for Data Center © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Tenant A Tenant B Tenant C Physical Network Infrastructure Abstract network view for an tenant • Decoupled from physical infra • Composed as a set of logical network resources

Slide 9

Slide 9 text

Next Gen Data Center Networking • Trend 1 • Disaggregation and White box • Trend 2 • Virtualization, Overlays, and OpenStack • Trend 3 • Two-stage Leaf-spine Clos- Fabrics with ECMP and Pods • Trend 4 • SDN, Policy, and Intent • Trend 5 • Big Data and Analytics 8/26/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com 9 Ref: https://www.linux.com/blog/event/open-networking-summit/2017/3/linux-foundation-highly-relevant-data-center-networking-evolution-says-sdxcentral-report

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Open Compute Project, OCP • Founded in 2011 • Global community for Open IT hardware • Increased Flexibility • Push for standard HW and Reduced Cost • Initial Data Center focus • Now broadening to telecom and Enterprise • Disaggregated • Fully open hardware with enabling software 8/26/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com 11

Slide 12

Slide 12 text

Slide 13

Slide 13 text

OCP, Networking • Fully disaggregated and open networking HW & SW • Operating System - Linux based operating systems & developer tools, and REST API’s • Fully automated configuration management & bare metal provisioning • Universal & Multi-Form Factor Switch motherboard hardware • Fully open integration & connectivity • Energy efficient power & cooling designs • Software Defined Networking (SDN) Ref: http://www.opencompute.org/wiki/Networking 13

Slide 14

Slide 14 text

Example: Wedge 100 Ref: https://code.facebook.com/posts/681382905244727/introducing-wedge-and-fboss-the-next-steps-toward-a-disaggregated-network/ Facebook Design CPU: Intel/ARM/… ASIC: Broadcom/Mellanox/… - NOS: Open Network Linux - Forwarding Agent: FBOSS - BMC: OpenBMC Hardware Software ”Switch as a Server” 14

Slide 15

Slide 15 text

OCP Networking - Software Switch Abstraction Interface, SAI − Defines API to provide a vendor-independent way of controlling forwarding elements, such as a switching ASIC, an NPU or a software switch in a uniform manner. Open Network Linux, ONL − Linux distribution (Debian) with added driver and configuration for running bare metal switches Open Optical Monitoring, OOM − Contents of optical module EEPROM accessible to python programmers. Open Network Install Environment, ONIE − Open “install environment” for bare metal network switches − ONIE enables a bare metal network switch ecosystem where end users have a choice among different network operating systems Ref: https://github.com/opencomputeproject 15

Slide 16

Slide 16 text

SDN Based Switch Models 8/26/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com 16 Traditional Switch Data Plane Control Plane Applications SDN-based System Data Plane Control Plane Applications Control Protocol Open Networking Switch Data Plane Linux OS Applications

Slide 17

Slide 17 text

Big Switch Networks Big Cloud Fabric

Slide 18

Slide 18 text

Shared ”One Big Switch” Architecture § Traditional Netframe Design § Single point of management § Proprietary, Vendor Lock-in, Fixed slots Big Cloud Fabric Controller Hierarchical Control Plane 1 3 Spine Switches 2 10G/40G Backplane 4 1 3 2 4 Leaf Switches 1G/ 10G/40G Workloads § Disaggregate Netframe to one “Big Switch” § Open, Centralized management Easy to scale-out your network

Slide 19

Slide 19 text

Overview of Big Cloud Fabric Open, Economical Solution for Existing Enterprise & Service Provider Data Center Multi-Orchestrated VM/Container Single Programmatic Interface for up to 64 Rack Fabric Big Switch Controller Full Automation for Provisioning, HA / Resiliency & Management Switch Light OS Open Network Linux (ONL) Based OS for Edgecore Networks switches OCP-enabled Switch High customizable & quality switch vendor Switch Light Virtual For OpenStack/OpenShift/ Kubernetes...etc deployments

Slide 20

Slide 20 text

Distributed Logical Routing Host 1 10.50.1.2 Host 2 10.50.1.3 10.50.1.0/24 Host 3 10.50.2.2 Host 4 10.50.2.3 10.50.2.0/24 Logical Network (TENANT T1) Rack 1 Rack 2 Spine Router IP 10.50.1.1 Router IP 10.50.2.1 Segment Green 10.50.2.0/24 Segment Orange 10.50.1.0/24 Host 1 10.50.1.2 Host 3 10.50.2.2 Host 2 10.50.1.3 Host 4 10.50.2.3 Physical Network Logical Tenant Router T1

Slide 21

Slide 21 text

Test Path Visibility Network Troubleshooting Spine 3 Spine 1 Spine 2 R1L2 R1L1 R3L2 R3L1 R2L2 R2L1 Mesos-agent-2 7Xkf8don6Y Spine Leaf Mesos-agent-3 nmDh0cpymd Ethernet 26 Ethernet 49 Ethernet 5 Ethernet 2 Ethernet 50 Ethernet 19 enp5s0f1 enp8s0f0 qvo7Xkf8don6Y qvonmDh0cpymd vSwitch Source Container BIG CLOUD FABRIC CONTROLLER (CLI, GUI, API) Destination Container One-click flow trace across the fabric No box-by-box hopping

Slide 22

Slide 22 text

ONF Trellis

Slide 23

Slide 23 text

Metro Router White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box Open Source SDN-based Bare-metal White Box White Box R,E,M- Access CORD Architecture 23 ONOS Controller Cluster vRouter Control XOS (Orchestrator) vSG VNF VNF VNF VNF vSG VNF VNF VNF VNF vSG VNF VNF VNF VNF OVS OVS OVS OVS OVS Residential Mobile Enterprise Underlay Control Underlay Overlay Control Overlay vOLT Control

Slide 24

Slide 24 text

What is Trellis? 24 Datacenter Leaf-Spine Fabric Underlay Virtual Network Overlay Unified SDN Control Of Underlay & Overlay ONOS Controller Cluster & Apps Trellis is the enabling Network Infrastructure for CORD Trellis Provides Common control over underlay & overlay networks, including 1. Service Composition for Tenant Networks 2. Distributed Virtual Routing 3. Optimized Delivery of Multicast Traffic Streams

Slide 25

Slide 25 text

Underlay Fabric – Open Hardware 25 White Box SDN Switch Edgecore AS6712-32x Spine Switch 32 x 40G ports downlink to leaf switches 40G QSFP+/DAC GE mgmt. AS6712-32x White Box SDN Switch Edgecore AS6712-32x Leaf Switch 24 x 40G ports downlink to servers and vOLT 8 x 40G ports uplink to different spine switches ECMP across all uplink ports GE mgmt. AS6712-32x

Slide 26

Slide 26 text

26 BRCM ASIC OF-DPA Indigo OF Agent OF-DPA API OpenFlow 1.3 Leaf/Spine Switch Software Stack OCP Software - ONL ONIE OCP Bare Metal Hardware BRCM SDK API ONOS OCP: Open Compute Project ONL: Open Network Linux ONIE: Open Network Install Environment BRCM: Broadcom Merchant Silicon ASICs OF-DPA: OpenFlow Datapath Abstraction ONL-2.0.0-ONL-OS-DEB8-2016-12-22 OF-DPA 3.0 EA4 ONOS 1.8.9 CORD-3.0 Underlay Fabric – Software Stacks

Slide 27

Slide 27 text

L2 Unicast 27 Leaf1 Leaf2 Spine1 Spine2 Host1 Host2 Host3 OLT Upstream Router Quagga ONOS

Slide 28

Slide 28 text

L2 Broadcast 28 Leaf1 Leaf2 Spine1 Spine2 Host1 Host2 Host3 OLT Upstream Router Quagga ONOS

Slide 29

Slide 29 text

L3 Unicast 29 Leaf1 Leaf2 Spine1 Spine2 Host1 Host2 Host3 OLT Upstream Router Quagga ONOS

Slide 30

Slide 30 text

L3 Multicast 30 Leaf1 Leaf2 Spine1 Spine2 Host1 Host2 Host3 OLT Upstream Router Quagga ONOS

Slide 31

Slide 31 text

vRouter Integration 31 Leaf1 Leaf2 Spine1 Spine2 Host1 Host2 Host3 OLT Upstream Router Quagga ONOS BGP Data

Slide 32

Slide 32 text

vSG Integration 32 Leaf1 Leaf2 Spine1 Spine2 Host1 Host2 Host3 OLT Upstream Router Quagga ONOS Q-in-Q

Slide 33

Slide 33 text

Put everything all together… 33 Leaf1 Leaf2 Spine1 Spine2 Host1 Host2 Host3 OLT Upstream Router Quagga ONOS

Slide 34

Slide 34 text

Virtual Network Overlay 34 OVS OVS OVS OVS OVS OVS OVS OVS OVS Service VNFs & vNets Non-overlapping addresses Service B Virtual Network Tenant Green Virtual Network Overlapping address space Connectivity isolation VMs/Containers Service Y Virtual Network Tenant Blue Virtual Network Services can dynamically grow or shrink VXLAN Overlay VXLAN Overlay VXLAN Overlay Single VXLAN port in OVS

Slide 35

Slide 35 text

Trellis Summary Underlay Fabric • L2/L3 spine-leaf fabric – Bare-metal hardware + open source software • SDN control plane – No distributed protocols • Modern ASIC data plane – 1.28 Tbps switching bandwidth for each switch Virtual Network Overlay • Designed for NFV – ChainedVNFs using with best principles of cloud • Overlay Control – XOS and VTN implement service graph • OVS + VXLAN Data Plane Unified SDN Control • Common Control – Opportunity for optimized service delivery 35

Slide 36

Slide 36 text

Slide 37

Slide 37 text

FreeRangeRouting • IP routing protocol suite for Linux and Unix platforms • Includes protocol daemons for BGP, IS-IS, LDP, OSPF, PIM, EIGRP and RIP • Seamless integration with native Linux/Unix IP networking stacks including connecting hosts / VMs / containers • Fork from Quagga • Community driven based on • Github • Mail List • Slack 8/26/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com 37 Ref: https://frrouting.org/

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Summary • Hardware and Software Disaggregation • Unified Centralized Control and Management • Flexible • Security • Visibility • Lower CAPEX and OPEX • Deliver new service quickly and efficiency 8/26/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com 43 "You disaggregate to get choice; you aggregate to get efficiencies"

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Appendix • Software for Open Networking in the Cloud SONiC • http://azure.github.io/SONiC/ • Ecosystem momentum positions Microsoft’s Project Olympus as de facto open compute standard • https://azure.microsoft.com/en-us/blog/ecosystem-momentum-positions-microsoft-s- project-olympus-as-de-facto-open-compute-standard/ • Channel 9 - Microsoft Project Olympus • https://channel9.msdn.com/Series/Microsoft-Global-Datacenters/Microsoft-Project- Olympus • GitHub – Project Olympus • https://github.com/opencomputeproject/Project_Olympus 8/26/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com 45