Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OpenSolaris Network

March 17, 2013

OpenSolaris Network


March 17, 2013

More Decks by fraosug

Other Decks in Technology


  1. Pre - S10 TCP/IP Stack Pre - S10 TCP/IP Stack

    Solaris 2.x TCP/IP stack Uses STREAMS horizontal perimeter Setting up a STREAMs was expensive Connections were more long lived (NFS, ftp, etc.), Amortization over the life of the connection The world has changed: Web applications generated packets of heavy loads Multimedia and other real-time application require Real-time Constant-Rate Low-Latency Service Multimedia mostly datagram based connection No flow control
  2. Solaris must deal with higher short-lived connections Servers became heavily

    MP/CMT and NUMA centric running large number of CPUs/Cores. STREAMs by design did not have any CPU affinity packets for a particular connections moved around to different CPU. The cost of context switching became high Solaris needed to move away from STREAMs architecture Pre - S10 TCP/IP Stack Pre - S10 TCP/IP Stack
  3. Nemo – New-Architecture Nemo – New-Architecture Virtualization of the Layer

    two of the Network Stack Performance improved: Kernel clients avoid using DLPI Avoid DLPI within the DLS and MAC modules VLAN Implementation
  4. Nemo Nemo • The Nemo architecture: •Provides DLPI interface to

    the network layer services •Furthermore it provides a performance boost •Function call-based interface. •It virtualizes the Data Link layer of the network stack •No longer a one-to-one correspondence network interfaces and devices.
  5. IP Instances IP Instances • Provide a more robust architecture

    for zones •networking •Provides per-zone routing tables •Provides per-zone ARP table •i.e. provide an instance of IP per-zone
  6. • Shared stack •Uses global routing table •Global interface management,

    performance tuning, etc •Short-cut routing •Packet filtering applied to the entire machine • Exclusive Instances •Routing table per zone •Each zone can tweak TCP, etc, settings in its own way •Each zone decides what filtering it wants IP Instances - Zones Networking IP Instances - Zones Networking S hared vs Exclusive Instances
  7. • An exclusive instance cannot be managed from the global

    zone • Local zone root has full control over IP •Use of ndd to tune IP is allowed and is private • Network interfaces delegated for exclusive use are not visible in the global zone S hared vs Exclusive Instances IP Instances - Zones Networking IP Instances - Zones Networking
  8. IP Instances – Changes to security • Privilege split of

    SYS_NET_CONFIG •SYS_IP_CONFIG (new) for zones • Can snoop from inside a zone •But can still snoop using the interface in global too! • Security Threat •Zone can forge ethernet packets IP Instances - Zones Networking IP Instances - Zones Networking
  9. • Stack and NIC Virtualization • Resource partitioning, QoS/Diffserv (without

    performance penalties) • SLA on a per connection basis • Better Defense against DDOS attacks • Real time usage and history • N2 performance > Polling on forwarding path (performance) > S/W fanout to multiple cores (utilization) • Class of service support • ISV support (APIs for configuration, statistics, traps, etc) • Network Device Consolidation Crossbow Features Crossbow Features
  10. Technical Obstacles Technical Obstacles • Obstacles to achieving network virtualization:

    • Network processing in interrupt context • Anonymous packet processing in kernel • Common queues • Performance can be degraded by the extra processing to enforce fairness, resource control or network virtualization • No isolation for flows
  11. • Divide NIC memory, DMA channels, etc and use a

    flow classifier to build a virtual stack on each H/W partition • Each Virtual NIC is owned by the FireEngine Squeue's which independently switch the VNIC between interrupt & polling mode • Rate of packet arrival from a VNIC is independently controlled by the Squeue owning the VNIC The Crossbow Architecture The Crossbow Architecture
  12. • Sample mpstat output Mpstat (older driver) intr ithr csw

    icsw migr smtx srw syscl usr sys wt idl 10818 8607 4558 1547 161 1797 289 19112 17 69 0 12 Mpstat (GLDv3 based driver) intr ithr csw icsw migr smtx srw syscl usr sys wt idl 2823 1489 875 151 93 261 1 19825 15 57 0 27 • Notice the decrease in interrupts, context switches, mutex contentions, etc. and increase in idle time • Crossbow allows each VNIC to be dynamically polled by its Squeue Effects of Dynamic Polling Effects of Dynamic Polling
  13. •Setting up a Virtual network •within a single box –

    automatically •Bourne Shell scripts suitable for setting up: •Demo - •Tests - •Jumpstart - Environments •Tested with OpenSolaris SXCE Build 130 • site.xml did not work to enable/disable Services • OpenSolaris some issues Virtual Network – Practical Example Virtual Network – Practical Example
  14. -rwxrwxrwx 1 root root 2113 2010-05-25 11:22 cl_crossbow.sh -rwxrwxrwx 1

    root root 142 2010-05-22 21:24 exec_cl_crossbow.sh -rwxr-xr-x 1 root root 194 2010-05-25 13:04 exec_nv_mk_crossbow.sh -rwxr-xr-x 1 root root 194 2010-05-25 13:04 exec_os_mk_crossbow.sh -rwxr-xr-x 1 root root 14120 2010-05-25 16:42 nv_mk_crossbow.sh -rwxrwxrwx 1 root root 12974 2010-05-25 12:06 os_mk_crossbow.sh -rw-r--r-- 1 root root 26816 2010-05-25 11:14 site.xml -rwxrwxrwx 1 root root 4120 2010-05-25 12:13 var.init Virtual Network – Practical Example Virtual Network – Practical Example List of the shell scripts