Page 1 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   GPU-Accelerated VDI on OpenStack OpenStack DC MeetUp June 18, 2013 Nimbis Services, Inc. http://www.nimbisservices.com Brian Schott, CTO [email protected] 

Page 2 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   About Nimbis Services Nimbis Services is lowering the barriers to high- performance computing (HPC) and offering a secure marketplace for technical computing in the cloud. Nimbis partners with leading HPC & cloud providers, software vendors, and domain experts to configure and broker secure on-demand pay-as-you-go access to applications. 

Page 3 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Challenges for GPU-Accelerated Virtual Desktop Infrastructure (VDI) on OpenStack §  Making the GPU available to a virtual machine. (3 basic approaches, ok, maybe 4). Ø  Bare Metal / Containers (lxc) Ø  PCI Pass-Through Ø  Para-virtualization (proxy device drivers) Ø  Remote Display Protocols (X, NX, VirtualGL, WebGL) §  Remote desktop protocol that supports GPU acceleration. Ø  Host and Guest Combinations: Linux or Windows Ø  Remote Clients: Linux, Windows, OSX, Web Browser, Tablet Ø  Networking: “The Joys of Firewalls”, or “Any Port as Long as it is 443” Ø  Licenses: “This is too good to be free.” §  Provisioning GPU resources from within OpenStack. Ø  Nova compute virtualization driver (plugin). Ø  GPU instance types, resource discovery, and scheduling. Ø  This is still under development in the community. Ø  Full disclosure: Nimbis hand-deployed the GPU instances for testing. 

Page 4 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Nimbis Testbed Hardware §  “Production” Cisco System Ø  (1) UCS C220 M3 – dual quad-core 3.3GHz Intel Xeon E5-2643, 192GB SDRAM, 2TB SAS-7K, dual channel 10GbE interface. Ø  (2) UCS C220 M3 – dual quad-core 2.4GHz Intel Xeon E5-2609, 192GB SDRAM, 2TB SAS-7K, dual channel 10GbE interface. Ø  (3) UCS C240 M3 – dual quad-core 2.4GHz Intel Xeon E5-2609, 192GB SDRAM, 6TB SAS-7K, dual-channel 10GbE. •  (1) Nvidia GRID K2 Board Ø  (2) Nexus 3064 10GbE Switches Ø  (2) ASA 5520 Adaptive Security Appliances Ø  (2) Cisco Catalyst 2960S 1GbE switches §  “Development” Cisco System Ø  (3) Cisco UCS C200 M2 Ø  (1) Nexus 3064 10GbE Switch Ø  (1) ASA 5520 Adaptive Security Appliance Ø  (1) Cisco Catalyst 2960S 1GbE switch Cores: 72 Memory: 1440 GB Disk: 36 TB Servers: 9* Switches: 9* * What do you expect with a Cisco testbed? 

Page 5 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Nvidia GRID GPU Virtualization (Kepler Architecture) Source: http://www.nvidia.com/object/grid-boards.html Source: 1http://www.nvidia.com/content/PDF/kepler/NV_DS_Tesla_KCompute_Arch_May_2012_LR.pdf GPU Virtualization¹ GRID boards feature the NVIDIA Kepler architecture that, for the first time, allows hardware virtualization of the GPU. This means multiple users can share a single GPU, improving user density while providing true PC performance and compatibility. 

Page 6 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   GPU-Accelerated VDI Approaches Considered §  Customer Requirements Ø  Combination of Windows and Linux guest images. Ø  Applications using OpenGL on both Windows and Linux. Ø  Variety of Windows (95, 7, 8) clients, OSX clients, and web browsers (IE7, IE8, Firefox, Chrome, Safari). Ø  Variety of firewall and IT/FSO software challenges. §  Technologies Tested Ø  Microsoft Hyper-V and RemoteFX Ø  XenServer with VNC / RDP / Guacamole Ø  Open Source Xen (libvirt) on Ubuntu Linux with VNC §  Possible Alternatives Ø  Linux Containers (lxc) •  USC/ISI team has the blueprint and code targeted for Havana. •  No Windows support. Ø  XenDesktop with HDX 3D Pro (not tested) •  Requires a custom client, Citrix Receiver, OpenGL support unclear. •  Does not integrate with OpenStack. 

Page 7 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Microsoft Hyper-V and RemoteFX §  Technology Ø  Windows Host GPU device with para-virtualized driver for windows guests. §  Host / Hypervisor Ø  Microsoft Windows Server 2012 Datacenter Edition (RemoteFX license). Ø  Latest Nvidia driver for Windows. §  Guests Ø  No GPU pass-through available or driver for Linux guests. Ø  Accelerates Windows DirectX calls, does software OpenGL rendering. §  Clients Ø  Good performance with standard Windows RDC > 8.0. Ø  Good performance from FreeRDP* OSX client, not Microsoft client. Ø  iOS and Android? Not tested, but FreeRDP clients exist on both. §  OpenStack Status Ø  Integrates with OpenStack Hyper-V nova compute drivers. Ø  Nova scheduler support for Hyper-V extra specs not on blueprint radar. Ø  Possible to use today with custom nova scheduler plugin. * http://www.freerdp.com 

Page 8 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Hyper-V and RemoteFX Benchmarks with Nvidia GRID K2 GFXBench)Test Metric RDP)Without) RemoteFX NVidia) Quadro)600) (~$150)* RDP)With) RemoteFX NVidia) Quadro)K5000) (~$1800)* Spark)Lines Fill$rate$)$Offscreen$(1080p) M$Texels/s 76 3291 34507 39776 Fill$rate$)$Onscreen M$Texels/s 60 2324 8760 7949 GFXBench$2.5$Egypt$HD$C24Z16$)$Offscreen$(1080p) Frames 124 4274 6610 70733 GFXBench$2.5$Egypt$HD$C24Z16$)$Offscreen$(1080p) Frames$/$s 1.1 76.3 58.5 626 GFXBench$2.5$Egypt$HD$C24Z16$)$Onscreen Frames 74 2622 4682 6776 GFXBench$2.5$Egypt$HD$C24Z16$)$Onscreen Frames$/$s 0.7 46.8 41.4 60 GFXBench$2.7$T)Rex$HD$C24Z16$)$Offscreen$(1080p) Frames 26 3124 3719 24434 GFXBench$2.7$T)Rex$HD$C24Z16$)$Offscreen$(1080p) Frames$/$s 0.5 55.8 66.4 436.3 GFXBench$2.7$T)Rex$HD$C24Z16$)$Onscreen Frames 15 4274 2708 3358 GFXBench$2.7$T)Rex$HD$C24Z16$)$Onscreen Frames$/$s 0.3 76.3 48.4 60 GFXBench$2.7$T)Rex$HD$Fixed$Timestep$)$Offscreen$(1080p) ms 1199830 56022 9117 1546 GFXBench$2.7$T)Rex$HD$Fixed$Timestep$)$Offscreen$(1080p) Frames$/$s 0.5 56022 61.4 1546 GFXBench$2.7$T)Rex$HD$Fixed$Timestep$)$Onscreen ms 1699985 10 11734 362.2 GFXBench$2.7$T)Rex$HD$Fixed$Timestep$)$Onscreen Frames$/$s 0.3 6761 47.7 9361 Triangle$throughput:$Textured$)$Offscreen$(1080p) M$Triangles/s 1 59.8 2209 59.8 Triangle$throughput:$Textured$)$Onscreen M$Triangles/s 0 303 6246 2098 Triangle$throughput:$Textured,$fragment$lit$)$Offscreen$(1080p) M$Triangles/s 1 300 1770 2049 Triangle$throughput:$Textured,$fragment$lit$)$Onscreen M$Triangles/s 0 303 4263 1678 Triangle$throughput:$Textured,$vertex$lit$)$Offscreen$(1080p) M$Triangles/s 1 299 1780 1665 Triangle$throughput:$Textured,$vertex$lit$)$Onscreen M$Triangles/s 0 300 7310 1672 Host: Microsoft Windows Server 2012 Datacenter Edition, Guest: Microsoft Windows Server 2012 Standard Evaluation, Client: FreeRDP on Mac OSX * http://gfxbench.com/ 

Page 9 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   XenServer with VNC / RDP / Guacamole §  Technology Ø  GPU pass-through of Nvidia GRID K2 to Windows and Linux guests. Ø  Two GPU devices available for two separate instances. Ø  Physical GPU virtualization drivers promised by end of the year (n=64?). §  Host / Hypervisor Ø  XenServer 6.1 Platinum Edition (enables GPU license). §  Guests Ø  Linux: specify Windows 7 as server template then standard CentOS 6.4 install process. Good GPU performance using VNC and VirtualGL. Ø  Windows: tried both RDP and TigerVNC do software OpenGL rendering. •  Unable to VNC or RDP to “console” display, possibly no monitor? §  Clients Ø  Windows and OSX Clients comparable to remote desktop to native boxes. Ø  Tried RDP, VNC, and Guacamole (HTML5 VNC client). §  OpenStack Status §  XenServer Integrates with OpenStack using XenServer plugins. §  Nova scheduler support for XenServer not on blueprint radar. 

Page 10 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Open Source Libvirt/Xen with VNC / RDP / Guacamole §  Technology Ø  GPU pass-through of Nvidia GRID K2 to Windows and Linux guests. Ø  Two GPU devices available for two separate instances. Ø  Unsure of GPU virtualization drivers status for n > 2. §  Host / Hypervisor Ø  Ubuntu 12.04, Xen (Xen hypervisor 4.1.0), libvirt (0.98) Ø  Xen HVM guest cfgfile in /etc/xen/ to enable VGA passthrough: gfx_passthru=1 pci=['yy:zz.n’] §  Guests Ø  Linux: Tested CentOS 6.4 image with latest Nvidia drivers (310.44). Verified device functioned. Expect similar results to XenServer. Ø  Windows: Did not test an image. Expect similar results to XenServer. §  OpenStack Status §  Xen supported within OpenStack using nova libvirt driver. §  Nova specs support for libvirt exists (i.e. this host has a GPU). §  Feasible to have support included in H or I release given lxc work. http://wiki.xen.org/wiki/Xen_VGA_Passthrough 

Page 11 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   XenServer Linux Virtual Machine Installation Hack 

Page 12 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Specify GPU Type 

Page 13 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   VirtualGL Architecture (VirtualGL.org) 

Page 14 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   GLXINFO for Linux Instance on XenServer 

Page 15 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Guacamole HTML5 Browser VNC 

Page 16 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Windows Server 2012 on XenServer 

Page 17 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Heterogeneous Architecture Support Blueprints https://wiki.openstack.org/wiki/ HeterogeneousInstanceTypes https://wiki.openstack.org/wiki/ ScheduleHeterogeneousInstances https://wiki.openstack.org/wiki/ HeterogeneousGpuAcceleratorSupport 

Page 18 GPU-Accelerated VDI on OpenStack NIMBIS  SERVICES,  INC.   Summary §  GPU-accelerated VDI pools within OpenStack are quite feasible. Ø  Code support is very rough around the edges, not production ready. Ø  We really need to get open community support organized for this feature. Ø  Several vertical vendor solutions out there, but Xen and KVM baselines should be possible as open-source. §  Questions?