Open Networking Phil Huang 黃秉鈞 [email protected] [email protected] Engineering & Business 

2 Sponsor 

Who am I? Phil Huang • Edgecore Open Networking Solution Engineer • ONOS/CORD Ambassador • SDNDS-TW Moderator Ref: https://www.linkedin.com/in/phil-huang-09b09895 3 

SDNDS-TW Ref: https://www.facebook.com/groups/sdnds.tw 4 

Blog Ref: http://blog.pichuang.com.tw 5 

6 Ref: https://www.sdxcentral.com/event/open-source-networking-orchestration-poc-production/ 

Open Compute Project Take control of your technology future 7 

8 Ref: https://www.sdxcentral.com/event/open-source-networking-orchestration-poc-production/ 

Facebook Datacenter Ref: http://www.zdnet.com/pictures/facebooks-data-centers-worldwide-by-the-numbers-and-in-pictures/ 9 

Open Compute Project, OCP • Founded 2011 • Global community for Open IT hardware Hardware Management Networking Data Center HPC Open Rack Server Storage Telco Ref: http://www.opencompute.org/ 10 

OCP, Networking • Fully disaggregated and open networking HW & SW • Operating System - Linux based operating systems & developer tools, and REST API’s • Fully automated configuration management & bare metal provisioning • Universal & Multi-Form Factor Switch motherboard hardware • Fully open integration & connectivity • Energy efficient power & cooling designs • Software Defined Networking (SDN) Ref: http://www.opencompute.org/wiki/Networking 11 

Example: Wedge 100 Ref: https://code.facebook.com/posts/681382905244727/introducing-wedge-and-fboss-the-next-steps-toward-a-disaggregated-network/ Facebook Design CPU: Intel/ARM/… ASIC: Broadcom/Mellanox/… - NOS: Open Network Linux - Forwarding Agent: FBOSS - BMC: OpenBMC Hardware Software ”Switch as a Server” 12 

OCP Networking - Software • Switch Abstraction Interface, SAI • Defines API to provide a vendor-independent way of controlling forwarding elements, such as a switching ASIC, an NPU or a software switch in a uniform manner. • Open Network Linux, ONL • Linux distribution (Debian) with added driver and configuration for running bare metal switches • Open Optical Monitoring, OOM • Contents of optical module EEPROM accessible to python programmers. • Open Network Install Environment, ONIE • Open “install environment” for bare metal network switches • ONIE enables a bare metal network switch ecosystem where end users have a choice among different network operating systems Ref: https://github.com/opencomputeproject 13 

White Box Switch 14 

White Box Switch&Server Business Route ODM/OEM Vendor Traditional Customer Brand Vendor Cloud Provider / Carrier 15 

DC Core Data Center Clos Fabric Cloud Service Providers Telecom Service Providers Enterprise & Campus IXP Ref: https://speakerdeck.com/pichuang/introduction-to-opennetworking 16 

Building on Bare Metal Compute Network U-Boot & ONIE BIOS & PXE Applications Network OS Linux, Windows… Applications Applications Bare Metal Server Bare Metal Switch Hardware U-Boot & ONIE BIOS & PXE Boot & Provision BIOS & PXE U-Boot & ONIE OS Network OS Linux, Windows… 17 

Ref: http://onie.org/ 18 

Ref: https://www.slideshare.net/CumulusNetworks/onie-cumulus-networks-webinar 19 

Ref: https://www.slideshare.net/CumulusNetworks/onie-cumulus-networks-webinar 20 

White Box = Hardware + ONIE + PowerPC, x86, ARM 21 

Open Networking = Apps + NOS + White Box 22 + 

Ref: https://www.slideshare.net/CumulusNetworks/onie-cumulus-networks-webinar 23 

ONIE – Installer Mode GRUB Menu 24 

Hardware Vendors Ref: https://github.com/opencomputeproject/onie/tree/master/machine 25 

Commercial Network OS Vendor • Cumulus Networks CumulusLinux • Linux based NOS at web-scale with configuration, provisioning & monitoring tools • Big Switch Networks BCF & BMF • Provide total solution Big Monitoring Fabric and Big Cloud Fabric via centralized controller • Pica8 PicaOS • Hybrid networking OpenFlow agent with native L2 and L3 features • IP Infusion OcNOS • Full function L2/L3 with MPLS/VPLS, Carrier Ethernet, OAM and Telecom features • Pluribus Network NetVisor • Full function L2/L3 with data analytics, service chaining and NFV function 26 

Open Source Software • Open Compute Project, OCP • Open Network Install Environment (ONIE): Universal NOS Loader • Switch Abstraction Interface (SAI): Standard HAL for silicon from multiple vendors • Open Optical Monitoring (OOM): Make the contents of optical mode EEPROM with python • Dell EMC OpenSwitch • Full featured Linux-based NOS • Engineered for disaggregated whitebox networking devices • SnapRoute FlexSwitch • L2 and L3 protocol suite, with comprehensive configuration APIs, automation and orchestration • OCP Accepted • Open Network Linux, ONL • OCP reference OS for switches • Forwarding Agents: OF-DPA & Indigo OF Agent, SAI, OpenNSL, FBOSS Agent, Quagga Routing • Open Network Foundation: Atrium SDN Distribution • OpenFlow physical (ONL with OF-DPA) and virtual switches • ONOS / ODL open-source OpenFlow controller, with BGP application based on Quagga • Microsoft Software for Open Networking in the Cloud, SONiC • Network OS components, including full function L3 routing • Contributed by Microsoft and co-contributors to OCP SONiC 27 

White Box Switch Models Traditional Switch Data Plane Control Plane Applications SDN-based System Data Plane Control Plane Applications Control Protocol Open Networking Switch Data Plane Linux OS Applications Ref: https://speakerdeck.com/pichuang/introduction-to-opennetworking 28 

29 

Open CORD Central Office Re-architected as a Datacenter 30 

31 Ref: https://www.sdxcentral.com/event/open-source-networking-orchestration-poc-production/ 

What is CORD? SDN + NFV + Cloud Open Source Software Commodity Hardware (Servers, White-Box Switches, I/O Blades) Large number of COs (4-5k) Each serves 10-100k Evolved over 40-50 years 300+ Types of equipment Huge source of CAPEX/OPEX CORD: Central Office Re-architected as a Datacenter Ref: https://wiki.opencord.org/display/CORD/CORD+Summit+--+July+29%2C+2016?preview=/1278537/1279414/Summit-Larry.pdf 32 

Vendor Lock-In Hardware 33 Server Trends X86 Hardware + Software Central Office (CO) 局端 

34 Overview of CORD 

35 CORD Architecture R,E,M- Access Metro Router vRouter Control Other App Other App XOS (Orchestrator) vSG vSG vSG VNF VNF VNF VNF VNF VNF VNF VNF VNF VNF VNF VNF OVS OVS OVS OVS OVS White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box White Box Open Source SDN-based Bare-metal White Box White Box Residential Mobile Enterprise Underlay Control Underlay Overlay Control Overlay ONOS Controller Cluster Control 

36 Current Teleco Central Office of AT&T Ref: http://www.netmanias.com/en/post/blog/10408/gigabit-internet-sdn-nfv/at-t-s-cord-the-ultimate-architecture-born-after-decades-of-innovation-in-the-communications-network CO: Central Office 

37 Disaggregated Optical Line Termination OLT: Optical Line Termination GPON: Gigabit Passive Optical Network 

AT&T Open GPON – Hardware • 48 Port, 1RU, I/O Pizza Box • GPON MAC • GPON protocol management • 802.1ad-compiant VLAN bridging • Ethernet MAC Ref: http://opencord.org/wp-content/uploads/2016/03/Virtual-OLT.pdf 38 

Hardware Software AT&T Open GPON – Software Ref: https://wiki.opencord.org/display/CORD/VOLTHA%3A+vOLT+Hardware+Abstraction 39 OpenFlow Controller Configuration Controller Ref: https://wiki.opencord.org/display/CORD/CORD+Summit+--+July+29%2C+2016?preview=/1278537/1279415/Tom%20Anschutz%20R-CORD%20Breakout.pdf 

40 Disaggregated OLT Hardware Data Plane 

What is Trellis? Datacenter Leaf-Spine Fabric Underlay Virtual Network Overlay Unified SDN Control Of Underlay & Overlay ONOS Controller Cluster & Apps Trellis is the enabling Network Infrastructure for CORD Trellis Provides Common control over underlay & overlay networks, including • Service Composition for Tenant Networks • Distributed Virtual Routing • Optimized Delivery of Multicast Traffic Streams Ref: https://wiki.opencord.org/display/CORD/Underlay+Fabric 41 

Trellis: CORD Network Infrastructure 42 

Fabric Hardware & Software Ref: https://wiki.opencord.org/display/CORD/Underlay+Fabric 43 

44 ON.LAB R-CORD of AT&T Ref: http://www.netmanias.com/en/post/blog/10408/gigabit-internet-sdn-nfv/at-t-s-cord-the-ultimate-architecture-born-after-decades-of-innovation-in-the-communications-network 

●Telecommunication hardware located at the home or business of a customer ●a.k.a Residential Gateway (RG) 45 L3 Customer Premises Equipment CPE: Customer Premises Equipment 

Access hardware - CPE •  Simple commodity NetGear device •  Flashed with OpenWrt •  Runs OVS as dataplane switch •  OpenFlow capable •  Runs 802.1X authentication •  Several design options available here •  OpenFlow enabled? •  Run a DHCP server? •  Should make use of IPv6? Ref: https://wiki.opencord.org/display/CORD/CORD+Summit+--+July+29%2C+2016?preview=/1278537/1279412/R-CORD%20disagg-Ali.pdf 46 

●Disaggregation and virtualization ○ L3 CPE Software move to NFVI at CO 47 virtual Subscriber Gateway (vSG) 

48 ON.Lab R-CORD Final Architecture 

OpenFlow Control Plan Protocol 49 

50 Ref: https://www.sdxcentral.com/event/open-source-networking-orchestration-poc-production/ 

SDN? OpenFlow? Ref: https://www.opennetworking.org/images/stories/downloads/sdn-resources/white-papers/wp-sdn-newnorm.pdf 51 

SDN Solution Overview 52 Ref: https://cumulusnetworks.com/blog/linux-sdn-networking/ 

Service Abstraction Layer/Core Boron: Platform for Network-Driven Business OpenFlow Enabled Devices Open vSwitches Additional Virtual & Physical Devices Data Plane Elements (Virtual Switches, Physical Device Interfaces) Controller Platform Services/Applications OVSDB NETCONF PCMM/ COPS SNBI LISP PCEP SNMP SXP Southbound Interfaces & Protocol Plugins OpenFlow USC CAPWAP OPFLEX Control Plane Functions OpenDaylight APIs REST/RESTCONF/NETCONF/AMQP Data Store (Config & Operational) Messaging (Notifications / RPCs) LACP AAA Authorization Filter Network Abstractions (Policy/Intent) Graphical User Interface Application and Toolkit (DLUX / NeXT UI) IoT Http/CoAP OF-Config Embedded Controller Applications Independent Network Applications • AAA • Hot Tracker • Infrastructure Utilities • L2 Switch • LISP Service • Link Aggregation Control Protocol • Open Flow Forwarding Rules Manager • OpenFlow Stats Manager • OpenFlow Switch Manager • Topology Processing • NetIDE • NetVirt • Neutron Northbound • OVSDB Neutron • SN Integration Aggregator • Service Function Chaining • Time Series Data Repository • Unified Secure Channel Mgr • User Network Interface Mgr • Virtual Tenant Network Mgr • Atrium Router • Cardinal • Centinel – Streaming Data Hdlr • Controller Shield • Deve Discovery, ID & Mgmt • DOCSIS Abstraction • Eman • Genius • NAT Application • ALTO Protocol Manager • Fabric as a Service • Group Based Policy Service • NEMO • Network Intent Composition OCP BGP Ref: https://wiki.opendaylight.org/view/Presentations 53 

ON.Lab ONOS BGP BMv2 ISIS Lisp Netconf OpenFlow OSPF OVSDB PCEP REST SNMP TL1 54 Ref: https://github.com/opennetworkinglab/onos/tree/master/protocols 

OpenFlow 55 Manage Flow Table via OpenFlow Messages 

SDN != OpenFlow != OpenNetworking 

OpenFlow Use Case -- Tapping https://www.youtube.com/watch?v=N7qPXGCS40E&list=PLnF6OjxgvUnzmJHrwZ251jsifecZm2Iv8 https://www.youtube.com/watch?v=8OWMoUOsJDE 57 

NetDevOps Network Automation with Configuration Management and Infrastructure as Code 58 

SDN Solution Overview 59 Ref: https://cumulusnetworks.com/blog/linux-sdn-networking/ 

Traditional Networking Methodology 60 • Logging in manually with a password • Typing commands • Copy and paste over console or terminal • Checking box by box 

61 Step 1: Using Git & GitHub § Configuration file repository § File change and revision management § Build for teams to work the same files 

62 Step 2: Commit Your Configuration $ git add . $ git commit -a -m "Add new switch configuration" –s [master 8cf8a38] Add new switch configuration 1 file changed, 1 insertion(+) create mode 100644 ofswitch.conf $ git push -u origin master 

63 Show Changes In Action 

64 Step 3: Choose CM 

65 Ansible Network Modules Ref: http://docs.ansible.com/ansible/list_of_network_modules.html 

66 Step 4: Templatize Configuration Files • Let you reuse code snippets • Avoid human error and typos 

67 Step 5: Virtualize Copy of Network 

68 Step 6: Continuous Integrations System • Catch error or typos early • Make sure all of the setting is same as production 

Design Unit Test and End-to-End Testing 69 

Open Source Routing Still Quagga in the future? 70 

71 Ref: https://www.sdxcentral.com/event/open-source-networking-orchestration-poc-production/ 

72 

1 FRR - A new Quagga fork with a more open development Martin Winter [email protected] 73 Ref: https://frrouting.org/ 

What is FRR ? ‣ Open Source (GPLv2+) Routing Stack ‣ Implements RIP, RIPng, OSPF (v2&v3), ISIS, BGP, PIM, LDP ‣ Fork of Quagga ‣ Works on Linux and most BSD based systems ‣ For use in many Clouds as virtual routers, white box vendors and network providers (full routing stack) 2 (for the not so technical People) 74 

FRR - Why a new fork? 3 Open Development Model Faster Development Community Driven 75 

76 

FRR - What’s different? ‣ Methodical vetting of submissions ‣ More automated testing of contributions ‣ Github centered development ‣ Elected Maintainers & Steering Committee ‣ Common Assets held in trust by Linux Foundation 5 77 

FRR – Current Status 78 Ref: https://github.com/FRRouting/frr/wiki/FRR-2.0-%E2%86%92-FRR-3.0 

• Open Networking 是趨勢，並正在發生 • Open Source + Commercial • 跨領域整合需求大幅增加，但只賣單一領域的產品 • Total Solution / Turnkey Solution • 自己掌握自己命運 • 網路工程師能力呈現 M 型化 • Coding 是必備技能 • 與 Community 協作能力越來越重要 • 跨領域能力要求更多 Conclusion 79 

Event – COSCUP 2017 80 Ref: https://www.facebook.com/groups/sdnds.tw/permalink/1362996730432839/ 

Event – SDNDS-TW Workshop 2017 81 Ref: https://hackmd.io/KwDgRgjAzAhgpnAtAEwEzGIgLKqB2RATgDY4BjRCABmLPmSyuSmSA===?view 

Event – NetDevOps: Network Automation with Ansible and Test Infra 82