Frankfurter OpenSolaris User Group Aklilu Hadish Senior Systems Engineer Sun Microsystems GmbH Germany / Langen 

Agenda: Solaris Networking Crossbow Network Virtualization IP- Instances ... Clearview GLDv3 (Nemo) FireEngine 

Pre - S10 TCP/IP Stack Pre - S10 TCP/IP Stack Solaris 2.x TCP/IP stack Uses STREAMS horizontal perimeter Setting up a STREAMs was expensive Connections were more long lived (NFS, ftp, etc.), Amortization over the life of the connection The world has changed: Web applications generated packets of heavy loads Multimedia and other real-time application require Real-time Constant-Rate Low-Latency Service Multimedia mostly datagram based connection No flow control 

Solaris must deal with higher short-lived connections Servers became heavily MP/CMT and NUMA centric running large number of CPUs/Cores. STREAMs by design did not have any CPU affinity packets for a particular connections moved around to different CPU. The cost of context switching became high Solaris needed to move away from STREAMs architecture Pre - S10 TCP/IP Stack Pre - S10 TCP/IP Stack 

Project FireEngine Project FireEngine 

FireEngine FireEngine 

FireEngine (continued ...) FireEngine (continued ...) 

FireEngine (continued ...) FireEngine (continued ...) 

FireEngine (continued ...) FireEngine (continued ...) 

FireEngine (continued ...) FireEngine (continued ...) 

FireEngine (continued ...) FireEngine (continued ...) 

Nemo - GLDv3 Nemo - GLDv3 

Nemo Nemo GLDv3 •Interrupt Moderation •VLAN •Trunking •IPMP •IP-Tunneling 

Pre-Nemo Pre-Nemo VLAN Implementation DLPI: Monolitic driver utilizing the Generic LAN Driver (GLD) 

Enter Nemo Enter Nemo 

Nemo – New-Architecture Nemo – New-Architecture Virtualization of the Layer two of the Network Stack Performance improved: Kernel clients avoid using DLPI Avoid DLPI within the DLS and MAC modules VLAN Implementation 

Nemo Nemo • The Nemo architecture: •Provides DLPI interface to the network layer services •Furthermore it provides a performance boost •Function call-based interface. •It virtualizes the Data Link layer of the network stack •No longer a one-to-one correspondence network interfaces and devices. 

Nemo – GLDv3 Interrupt Moderation Nemo – GLDv3 Interrupt Moderation 

Nemo – Interrupt Moderation Nemo – Interrupt Moderation 

Nemo - Trunking Nemo - Trunking 

Clearview Clearview 

Clearview Clearview 

Clearview Clearview 

Clearview - Vanity Naming: Clearview - Vanity Naming: 

Clearview - Vanity Naming Clearview - Vanity Naming 

Clearview – IP Tunneling Clearview – IP Tunneling GLDV3 Driver GLDV3 Driver 

IP Instances IP Instances 

IP Instances IP Instances • Provide a more robust architecture for zones •networking •Provides per-zone routing tables •Provides per-zone ARP table •i.e. provide an instance of IP per-zone 

IP Instances IP Instances 

• Shared stack •Uses global routing table •Global interface management, performance tuning, etc •Short-cut routing •Packet filtering applied to the entire machine • Exclusive Instances •Routing table per zone •Each zone can tweak TCP, etc, settings in its own way •Each zone decides what filtering it wants IP Instances - Zones Networking IP Instances - Zones Networking S hared vs Exclusive Instances 

• An exclusive instance cannot be managed from the global zone • Local zone root has full control over IP •Use of ndd to tune IP is allowed and is private • Network interfaces delegated for exclusive use are not visible in the global zone S hared vs Exclusive Instances IP Instances - Zones Networking IP Instances - Zones Networking 

IP Instances – Changes to security • Privilege split of SYS_NET_CONFIG •SYS_IP_CONFIG (new) for zones • Can snoop from inside a zone •But can still snoop using the interface in global too! • Security Threat •Zone can forge ethernet packets IP Instances - Zones Networking IP Instances - Zones Networking 

Crossbow Crossbow Network Network Virtualization Virtualization 

• Stack and NIC Virtualization • Resource partitioning, QoS/Diffserv (without performance penalties) • SLA on a per connection basis • Better Defense against DDOS attacks • Real time usage and history • N2 performance > Polling on forwarding path (performance) > S/W fanout to multiple cores (utilization) • Class of service support • ISV support (APIs for configuration, statistics, traps, etc) • Network Device Consolidation Crossbow Features Crossbow Features 

Technical Obstacles Technical Obstacles ● Obstacles to achieving network virtualization: ● Network processing in interrupt context ● Anonymous packet processing in kernel ● Common queues ● Performance can be degraded by the extra processing to enforce fairness, resource control or network virtualization ● No isolation for flows 

● Divide NIC memory, DMA channels, etc and use a flow classifier to build a virtual stack on each H/W partition ● Each Virtual NIC is owned by the FireEngine Squeue's which independently switch the VNIC between interrupt & polling mode ● Rate of packet arrival from a VNIC is independently controlled by the Squeue owning the VNIC The Crossbow Architecture The Crossbow Architecture 

Crossbow “Hardware Lanes” Crossbow “Hardware Lanes” 

Crossbow “Traffic Lanes” Crossbow “Traffic Lanes” Classifier HW Classifier HW Classifier SW Classifier SW 

● Sample mpstat output Mpstat (older driver) intr ithr csw icsw migr smtx srw syscl usr sys wt idl 10818 8607 4558 1547 161 1797 289 19112 17 69 0 12 Mpstat (GLDv3 based driver) intr ithr csw icsw migr smtx srw syscl usr sys wt idl 2823 1489 875 151 93 261 1 19825 15 57 0 27 ● Notice the decrease in interrupts, context switches, mutex contentions, etc. and increase in idle time ● Crossbow allows each VNIC to be dynamically polled by its Squeue Effects of Dynamic Polling Effects of Dynamic Polling 

Virtual Network Containers Virtual Network Containers 

Virtual Network – The Cloud Virtual Network – The Cloud 

Solaris Networking Features Availability Solaris Networking Features Availability 

Virtual Network – S etup Virtual Network – S etup 

•Setting up a Virtual network •within a single box – automatically •Bourne Shell scripts suitable for setting up: •Demo - •Tests - •Jumpstart - Environments •Tested with OpenSolaris SXCE Build 130 • site.xml did not work to enable/disable Services • OpenSolaris some issues Virtual Network – Practical Example Virtual Network – Practical Example 

-rwxrwxrwx 1 root root 2113 2010-05-25 11:22 cl_crossbow.sh -rwxrwxrwx 1 root root 142 2010-05-22 21:24 exec_cl_crossbow.sh -rwxr-xr-x 1 root root 194 2010-05-25 13:04 exec_nv_mk_crossbow.sh -rwxr-xr-x 1 root root 194 2010-05-25 13:04 exec_os_mk_crossbow.sh -rwxr-xr-x 1 root root 14120 2010-05-25 16:42 nv_mk_crossbow.sh -rwxrwxrwx 1 root root 12974 2010-05-25 12:06 os_mk_crossbow.sh -rw-r--r-- 1 root root 26816 2010-05-25 11:14 site.xml -rwxrwxrwx 1 root root 4120 2010-05-25 12:13 var.init Virtual Network – Practical Example Virtual Network – Practical Example List of the shell scripts 

More Information More Information http://opensolaris.org/os/project/crossbow http://opensolaris.org/os/project/clearview http://opensolaris.org/os/community/networking http://hub.opensolaris.org/bin/download/Project+ clearview/ WebHome/clearview-overview.pdf http://www.c0t0d0s0.org/archives/5355- Upcoming-Solaris- Features-Crossbow-Part-1-Virtualisation.html 

The Network is the The Network is the Computer Computer