304 KDDI Mobile Subs 66.9M Personal Mobile services Subs 31.1M 5G Sub penetration 64.2% FTTH Subs 5.4M IoT(Globally) 39.5M Telehouse DCs 40+ Locations Operating Revenue JPY 5,800B (USD 38.7B*) Total employees 49,659 ＊1USD=JPY150 KDDI has both mobile subs and fixed subs
304 Our telco private-cloud journey Fixed domain Common platform for Fixed domain (2nd Gen) Common platform 3rd Gen IaaS Fixed domain Mobile Domain OSS/BSS Domain Vertical NFV Vertical-NFV-Stack (1st Gen) Vertical-NFV- Stacks Mobile Fixed • Mobile prioritized stability with vertical integration • Fixed networks prioritized efficiency with horizontal integration • Currently, we are building a common platform for both systems 2020 2021 2024 Today’s speaker originally come from the fixed network team.
304 Our new 3rd gen deployment Back-born Network Global Rgion Keystone Horizon DNSaaS Region 1 Keystone Nova Neutron Cinder Etc. Region 2 Keystone Nova Neutron Cinder Etc. Region 3 Region 4 Deploy OpenStack clusters for each region Set up a global region to enable integrated use and provide features like authentication, DNSaaS, and dashboards consists of just under 2,000 serers
304 External Network General approach(Self-Service Net/NAT,Floating IP) For general (non-NFV) requirements, the use of NAT functionality and pre-provisioned provider networks allows for on-demand provisioning. Hypervisor vSwitch Leaf switch Leaf switch Spine switch Spine switch Spine switch Bonding VM Hypervisor vSwitch Bonding VM VM Since the overlay network cannot directly access external networks, it is common to use Neutron routers for SNAT or DNAT. Tunneling between virtual switches using VXLAN or Geneve allows for shared underlay networks, enabling on-demand provisioning. Performance Bad Agility Good Flexibility Bad
304 Datacenter network VM virtio SR-IOV VLAN VLAN VM virtio SR-IOV VLAN VLAN SVI SVI Traditional NFV approach(Provider Net for the tenant) Performance Good Agility Bad Flexibility Not Bad • OpenStack assigns VLANs to virtual NICs and SR-IOV. • Network configuration is separately handled by the datacenter network. • Performance is good due to datacenter network switching and forwarding. • Agility is reduced as OpenStack and the data center network are not integrated. The traditional approach is conservative, losing speed and flexibility, and failing to leverage the advantages of private cloud.
304 Wanted solution Traditional approch General approch Performance Flexibility Agility More balanced solution was needed (performance was often excessive)
304 Wanted solution Traditional approch General approch Wanted solution Performance Flexibility Agility More balanced solution was needed (performance was often excessive)
304 The concept of solution 1. High-performance switching can be achieved in ML2/OVS or ML2/OVN by utilizing OVS-DPDK. 2. Enable direct external communication from virtual networks created using overlay networks like VXLAN or Geneve. 3. Use routing protocols like BGP to inform external networks about networks created in OpenStack. 4. Using DPDK-implemented virtual routers achieves both performance and functionality.
304 Detail of the concept | 1/2 ToR Datacenter Network OpenStack virtual router Nova VM Nova VM ToR Datacenter Network OpenStack Nova VM Nova VM VLAN VLAN VLAN Overlay Routing protocols such as BGP Static/VRRP Static/VRRP VMs are L2-connected over overlay networks to a virtual router without NAT. The virtual router connects to the physical network using VLAN-type networks and BGP. Re-distribute the virtual network to the ToR as a connected route
304 Detail of the concept | 2/2 Connect each virtual router to each VRF separately and independently ToR VRF:Red OpenStack virtual router Nova VM Nova VM VLAN overlay Datacenter network virtual router Nova VM Nova VM VLAN overlay VRF:Yellow
304 Study of architecture ToR Datacenter network OpenStack Nova VM virtual router B. Single-stage/ Single vRouter Nova VM ToR Datacenter network OpenStack Nova VM virtual router A. Single-stage/ Multi vRouter Nova VM virtual router ToR Datacenter network OpenStack C. 2-Stage / Multi vRouter virtual router Nova VM virtual router Nova VM virtual router Considering scalability and operability, a two-stage Leaf/Spine Clos topology similar to datacenter networks is adopted. Bus connection Multiple interface Adopted
304 PoC Result ①Functionality/Performance Are there virtual routers that meet the use cases of NFV? • High-speed data plane using DPDK • Implementation of protocols like BGP • Stable software ②Operational Manageability How to manage an increasing number of virtual routers? • How to deploy them • Can they be automated? We conducted a PoC to verify feasibility and identified several issues. We are selected DPDK implemented virtual router appliance Main topics of this presentation
304 Managing virtual router Datacenter network OpenStack virtual router ToR ToR ToR virtual router virtual router Nova VM virtual router Nova VM virtual router Nova VM virtual router Nova VM virtual router Nova VM virtual router Admin-side Tenant-side BGP 169.254.0.0/22 .0.10 .0.11 .0.12 .0.124 .1.87 .1.84 .1.92 .3.23 ① We defined routers connected to the ToR as Admin-side routers and those in OpenStack networks as Tenant-side routers. ② Admin-side routers are statically managed, while Tenant-side routers are dynamically generated. We need a management strategy for them. We aim to manage Tenant-side routers via the Neutron API. Admin-side virtual routers, not dynamically increasing, are managed manually. Manage Tenant-side routers via Neutron.
304 • Handle all request • Select handling or not with flavor information in plugin • Handle the request of without flavor information Network Flavor and Service type framework Neutron/ServiceTypeFramework - OpenStack • Neutron has a mechanism to introduce plugins and branch processing for each network flavor. • It has long been used to introduce various plugins in ML2/OVS. • It is also implemented in ML2/OVN from Caracal and works with OVN. 2024.1 Series Release Notes — Neutron Release Notes documentation (openstack.org) Neutron/FlavorFramework - OpenStack Neutron API Normal OVN procedure Custom plugin procedure w/o Flavor All request
304 Operates like standard OpenStack On the surface, it behaves as part of OpenStack In the background: Launches virtual routers and applies configurations Shadow Resources Neutron Router DB Nova VM Neutron Net Custom Plugin Ansible Config Plugin Ansible Playbook Neutron API 1. Create Router VM/NW 2. Generate config And install config Utilize OpenStack's Neutron Service Type Framework / Network Flavor to create a custom Neutron plugin for virtual routers. Automate the process of launching Tenant-side router VMs and applying configurations with the openstack router create command. 3. Configure by Ansible
304 Provisioning Flow • General user(tenant) can view only interconnect network between Admin-side routers and Tenant-side routers each VRF • And request create router and attach the external-gateway $ openstack network list +-------------+-----------+-------------+ | ID | Name | Subnets | +-------------+-----------+-------------+ | 844de224-.. | VRF_Red | c8817d69-.. | | 386bd27f-.. | VRF_Green | da0b8169-.. | +-------------+-----------+-------------+ $ openstack router create ¥ --flavor-id xxxx ¥ my_router $ openstack router set ¥ --external-gateway VRF_Red my_router $ openstack router add port myrouter my_router_port User cannot view other tenant’s tenant-side vrouters and admin-side vrouters. User can create tenant-side vRouers and attach the external network as same of neutron’s router management manner Admin-side router Admin-side router Admin-side router Rouer-VM BGP VRF_Red 169.254.0.0/22 .0.10 .0.11 .0.12 .0.124 Internal network 192.168.0.0/24 Once request for API, our plugin create VM for virtual router appliance and configure router OS for advertising of internal network segment.
304 Generated shadow resources Mgmt Net Nova VM Name: rtvm-abcd-0 UUID: e3a8 Image: vrt-image Flavor: vrt-flavor Slot 3 Slot 4 Slot 5 rtvm-ext-trunk-port-abcd-0/ vrtvm-ext-trunk-abcd-0 rtvm-int-trunk-port-abcd-0/ Vrtvm-int-trunk-abcd-0 2001:db8::6e1a/128 rtvm-int-abcd rtvm-ext-abcd Nova VM Name: rtvm-abcd-1 UUID: 7d82 Image: vrt-image Flavor: vrt-flavor Slot 3 Slot 4 Slot 5 rtvm-ext-trunk-port-abcd-1/ vrtvm-ext-trunk-abcd-1 rtvm-int-trunk-port-abcd-1/ vrtvm-int-trunk-abcd-1 2001:db8::7d82/128 Router UUID: abcd Name: router-1 HA: True IF ext-gw Neutron routers are deployed as Nova VMs. Points 1. When HA is selected for the VM, it uses a two-node configuration with VRRP. 2. Each VM is created with three ports: a management port, an external port, and an internal port. The external and internal ports are trunked. 3. An empty network is created to connect only to the VMs for trunking. 4. Ports connected to the Neutron router are handled as VLAN additions to the trunk port. • Reason for using trunk ports: Due to router appliance constraints, vNIC hot-plugging was not supported, but adding/removing VLANs was.
304 Manage router configuration using a Ansible Neutron DB Custom Plugin/Python Module Database entries: • Network/CIDR information • Config of machine: vCPU? / HA? • IP address and other details Information not in the database is generated: • ASN/Router ID/etc. Ansible playbook is invoked from the Python Neutron module. Generation and application of configurations • The "desired state" is defined in the OpenStack database. • A declarative implementation defines the desired state as a configuration, rather than using procedural operations. Virtual Router
304 Additional Benefits: BGP between VM Admin-side Router Tenant-side Router VM ToR Overlay Overlay VLAN VRRP with virtual router Redistribute connected Admin-side Router Tenant-side Router VM ToR Overlay Overlay VLAN BGP with virtual router Redistribut connected + Received route from VM BGP speaker BGP BGP BGP BGP BGP • Tenant-side routers can speak BGP, allowing VM routing daemons (such as FRR) to communicate with VMs • This enables L3 virtual IP redundancy and IP anycast • MetalLB can also operate on Kubernetes with Tenant-side Router
304 L3 Connectivity Overview of combination with OVN IMPORTANT: Our plugin does not replace OVN Leverage the OVN: • L2 switching, L3 routing with NAT, ACL (Security Groups), and other functions are still using OVN • Only L3 non-NAT routing and dynamic routing use cases utilize our L3 plugin Virtual Machine eth0 Internal Network Standard Router eth1 Network Tenant-side Router eth2 Tenant-side Router Other Virtual Machine eth0 Network Port Security Port Security Port Security Port Security Our Plugin Function OVN Function Nova Function
304 Conclusion Conclusion and more • By utilizing traditional Neutron features (Network Flavor/Service Type) , new ML2/OVN functionalities, and DPDK virtual router appliances, KDDI have achieved a high-speed and flexible network for NFV use cases • This balances "Performance", "Agility", and "Flexibility", solving traditional issues • Our network solution and K8s on OpenStack enhances K8s's simple networking capabilities, enabling efficient K8s operations for CNF deployments in NFV SPECIAL THANKS • We express our gratitude to Red Hat for their cooperation in making this possible. • Additionally, we have jointly published a white paper with Red Hat about OpenShift on OpenStack configuration. White Paper: Building an efficient private telco cloud with OpenShift on OpenStack
tj8000rpm
sugamasao
estie
ewolff
lycorp_recruit_jp
chou500
salaboy
ysknsid25
koichiotomo
hogelog
oracle4engineer
kenjikondobai
ronnnnn
philhawksworth
speakerdeck
garrettdimon
tanoku
bryan
maggiecrowley
eitanlees
keavy
morganepeng
mclloyd
lara
davidbonilla
