Trading Flexibility for Performance: The HPC Story in OpenStack

Trading Flexibility for Performance: The HPC Story in OpenStack

8fbd28ad59a1aa317a5ec175b0778359?s=128

Stephen Finucane

May 27, 2019
Tweet

Transcript

  1. Trading Flexibility for Performance The HPC Story in OpenStack Stephen

    Finucane Software Engineer, Red Hat @stephenfin
  2. Photo by Jackson Jost on Unsplash

  3. $ openstack flavor create --vcpus 8 --ram 8192 --disk 0

    my-flavor $ openstack server create --flavor my-flavor my-server
  4. Photo by Pero Kalimero on Unsplash

  5. $ openstack flavor create --vcpus 8 --ram 8192 --disk 0

    my-flavor
  6. $ openstack flavor create --vcpus 8 --ram 8192 --disk 0

    my-flavor $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
  7. $ openstack flavor create --vcpus 8 --ram 8192 --disk 0

    my-flavor $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor $ openstack server create --flavor my-flavor my-server
  8. Photo by Josep Castells on Unsplash

  9. Photo by Paolo Nicolello on Unsplash

  10. #1 CPU Pinning

  11. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate

    \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
  12. node #0 CPU #0 CPU #1 CPU #2 CPU #3

  13. node #0 CPU #0 CPU #1 CPU #2 CPU #3

    $ openstack server create ...
  14. node #0 CPU #0 CPU #1 CPU #2 CPU #3

    $ openstack server create ... $ openstack server create ... $ openstack server create ...
  15. node #0 CPU #0 CPU #1 CPU #2 CPU #3

    $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ... $ openstack server create ...
  16. None
  17. Is it always?

  18. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate

    \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
  19. node #0 CPU #0 CPU #1 CPU #2 CPU #3

  20. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate

    \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
  21. #2 NUMA Awareness

  22. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate

    \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
  23. node #0 node #1 node #2 node #3 node #1

    CPU #0 CPU #1 CPU #2 CPU #3
  24. node A node B Local Access Remote Access Memory Channel

    Interconnect Memory Channel
  25. node A node B node C node D

  26. node A node B node C node D

  27. node #1 CPU #4 CPU #5 CPU #6 CPU #7

    node #0 CPU #0 CPU #1 CPU #2 CPU #3
  28. node #1 CPU #4 CPU #5 CPU #6 CPU #7

    node #0 CPU #0 CPU #1 CPU #2 CPU #3
  29. node #1 CPU #4 CPU #5 CPU #6 CPU #7

    node #0 CPU #0 CPU #1 CPU #2 CPU #3
  30. node #1 CPU #4 CPU #5 CPU #6 CPU #7

    node #0 CPU #0 CPU #1 CPU #2 CPU #3
  31. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate

    \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
  32. node #1 CPU #4 CPU #5 CPU #6 CPU #7

    node #0 CPU #0 CPU #1 CPU #2 CPU #3
  33. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate

    \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
  34. #3 Real Time

  35. Photo by Echo Grid on Unsplash

  36. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate

    \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
  37. Hardware Host OS/Hypervisor Guest OS

  38. 1. Disable advanced CPU features (HyperThreading, Power Management, Turbo Boost,

    …) 2. Install host dependencies (real-time kernel, real-time KVM module, real-time tuned host profiles, …) 3. Configure host memory and CPU parameters (enable hugepages, isolate some cores, …) 4. Install guest dependencies (real-time kernel, real-time tuned guest profiles, …)
  39. $ openstack flavor set \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate

    \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-5 \ --property hw:numa_cpus.1=6,7 \ --property hw:numa_mem.0=6144 \ --property hw:numa_mem.1=2048 \ --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 \ --property hw:mem_page_size=1GB \ my-flavor
  40. #1 CPU Pinning #2 NUMA Awareness #3 Real Time

  41. Photo by Paolo Nicolello on Unsplash

  42. NFV is complex.

  43. HPC is complex.

  44. So…?

  45. 1. Document it (make the existing stuff easier to understand)

    2. Improve it (extend existing features and add new ones as necessary) 3. Replace it (replace the existing features with easier-to-use features)
  46. Photo by Marcus Dall Col on Unsplash

  47. Trading Flexibility for Performance The HPC Story in OpenStack Stephen

    Finucane Software Engineer, Red Hat @stephenfin