Trading Flexibility for Performance: The HPC Story in OpenStack

Transcript

1. Trading Flexibility for Performance The HPC Story in OpenStack Stephen Finucane Software Engineer, Red Hat @stephenfin
2. Photo by Jackson Jost on Unsplash
3. $ openstack flavor create --vcpus 8 --ram 8192 --disk 0 my-flavor $ openstack server create --flavor my-flavor my-server
4. Photo by Pero Kalimero on Unsplash
5. $ openstack flavor create --vcpus 8 --ram 8192 --disk 0 my-flavor
6. $ openstack flavor create --vcpus 8 --ram 8192 --disk 0 my-flavor $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
7. $ openstack flavor create --vcpus 8 --ram 8192 --disk 0 my-flavor $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor $ openstack server create --flavor my-flavor my-server
8. Photo by Josep Castells on Unsplash
9. Photo by Paolo Nicolello on Unsplash
10. #1 CPU Pinning
11. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
12. node #0 CPU #0 CPU #1 CPU #2 CPU #3
13. node #0 CPU #0 CPU #1 CPU #2 CPU #3 $ openstack server create ...
14. node #0 CPU #0 CPU #1 CPU #2 CPU #3 $ openstack server create ... $ openstack server create ... $ openstack server create ...
15. node #0 CPU #0 CPU #1 CPU #2 CPU #3 $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ...
17. Is it always?
18. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
19. node #0 CPU #0 CPU #1 CPU #2 CPU #3
20. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
21. #2 NUMA Awareness
22. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
23. node #0 node #1 node #2 node #3 node #1 CPU #0 CPU #1 CPU #2 CPU #3
24. node A node B Local Access Remote Access Memory Channel Interconnect Memory Channel
25. node A node B node C node D
26. node A node B node C node D
27. node #1 CPU #4 CPU #5 CPU #6 CPU #7 node #0 CPU #0 CPU #1 CPU #2 CPU #3
28. node #1 CPU #4 CPU #5 CPU #6 CPU #7 node #0 CPU #0 CPU #1 CPU #2 CPU #3
29. node #1 CPU #4 CPU #5 CPU #6 CPU #7 node #0 CPU #0 CPU #1 CPU #2 CPU #3
30. node #1 CPU #4 CPU #5 CPU #6 CPU #7 node #0 CPU #0 CPU #1 CPU #2 CPU #3
31. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
32. node #1 CPU #4 CPU #5 CPU #6 CPU #7 node #0 CPU #0 CPU #1 CPU #2 CPU #3
33. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
34. #3 Real Time
35. Photo by Echo Grid on Unsplash
36. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
37. Hardware Host OS/Hypervisor Guest OS
38. 1. Disable advanced CPU features (HyperThreading, Power Management, Turbo Boost, …) 2. Install host dependencies (real-time kernel, real-time KVM module, real-time tuned host profiles, …) 3. Configure host memory and CPU parameters (enable hugepages, isolate some cores, …) 4. Install guest dependencies (real-time kernel, real-time tuned guest profiles, …)
39. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
40. #1 CPU Pinning #2 NUMA Awareness #3 Real Time
41. Photo by Paolo Nicolello on Unsplash
42. NFV is complex.
43. HPC is complex.
44. So…?
45. 1. Document it (make the existing stuff easier to understand) 2. Improve it (extend existing features and add new ones as necessary) 3. Replace it (replace the existing features with easier-to-use features)
46. Photo by Marcus Dall Col on Unsplash
47. Trading Flexibility for Performance The HPC Story in OpenStack Stephen Finucane Software Engineer, Red Hat @stephenfin