Tweet
Tweet

Slide 1

Slide 1 text

The Wonders of NUMA (Or Why Your High-Performance Application Doesn't Perform) Stephen Finucane OpenStack Software Developer 23rd May 2018

Slide 2

Slide 2 text

What is NUMA?

Slide 3

Slide 3 text

INSERT DESIGNATOR, IF NEEDED 3 What is NUMA? Non-Uniform Memory Architecture

Slide 4

Slide 4 text

INSERT DESIGNATOR, IF NEEDED 4 What is NUMA? UMA (Uniform Memory Access) Historically, all memory on x86 systems is equally accessible by all CPUs. Known as Uniform Memory Access (UMA), access times are the same no matter which CPU performs the operation. NUMA (Non-Uniform Memory Access) This behavior is no longer the case with recent x86 processors. In Non-Uniform Memory Access (NUMA), system memory is divided into zones (called nodes), which are allocated to particular CPUs or sockets. Access to memory that is local to a CPU is faster than memory connected to remote CPUs on that system.

Slide 5

Slide 5 text

INSERT DESIGNATOR, IF NEEDED 5 What is NUMA? node A node B Local Access Remote Access Memory Channel Interconnect Memory Channel

Slide 6

Slide 6 text

INSERT DESIGNATOR, IF NEEDED 6 What is NUMA? node A node B node C node D

Slide 7

Slide 7 text

INSERT DESIGNATOR, IF NEEDED 7 What is NUMA? node A node B node C node D

Slide 8

Slide 8 text

INSERT DESIGNATOR, IF NEEDED 8

Slide 9

Slide 9 text

NUMA in OpenStack

Slide 10

Slide 10 text

INSERT DESIGNATOR, IF NEEDED 10 NUMA in OpenStack ● NUMA Guest Topologies ● Guest vCPU Placement ● PCI NUMA Affinity ● vGPU, Neutron Network NUMA Affinity

Slide 11

Slide 11 text

INSERT DESIGNATOR, IF NEEDED 11 NUMA in OpenStack ● NUMA Guest Topologies ● Guest vCPU Placement ● PCI NUMA Affinity ● vGPU, Neutron Network NUMA Affinity

Slide 12

Slide 12 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 6 --ram 6144 --disk 20 test.numa 12 NUMA Guest Topologies

Slide 13

Slide 13 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 6 --ram 6144 --disk 20 test.numa $ openstack flavor set test.numa \ --property hw:numa_nodes=2 13 NUMA Guest Topologies

Slide 14

Slide 14 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 6 --ram 6144 --disk 20 test.numa $ openstack flavor set test.numa \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-3 \ --property hw:numa_cpus.1=4,5 \ --property hw:numa_mem.0=4096 \ --property hw:numa_mem.1=2048 14 NUMA Guest Topologies

Slide 15

Slide 15 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 6 --ram 6144 --disk 20 test.numa $ openstack flavor set test.numa \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-3 \ --property hw:numa_cpus.1=4,5 \ # guest vCPUs - not host CPUs --property hw:numa_mem.0=4096 \ --property hw:numa_mem.1=2048 15 NUMA Guest Topologies

Slide 16

Slide 16 text

INSERT DESIGNATOR, IF NEEDED 16 NUMA in OpenStack ● NUMA Guest Topologies ● Guest vCPU Placement ● PCI NUMA Affinity ● vGPU, Neutron Network NUMA Affinity

Slide 17

Slide 17 text

INSERT DESIGNATOR, IF NEEDED 17 NUMA in OpenStack ● NUMA Guest Topologies ● Guest vCPU Placement ● PCI NUMA Affinity ● vGPU, Neutron Network NUMA Affinity

Slide 18

Slide 18 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 test.pinned 18 Guest vCPU Placement

Slide 19

Slide 19 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 test.pinned $ openstack flavor set test.pinned \ --property hw:cpu_policy=dedicated 19 Guest vCPU Placement

Slide 20

Slide 20 text

INSERT DESIGNATOR, IF NEEDED 20 Guest vCPU Placement node #0 core #0 core #1 core #3 core #2 node #1 core #0 core #1 core #3 core #2

Slide 21

Slide 21 text

INSERT DESIGNATOR, IF NEEDED 21 Guest vCPU Placement node #0 core #0 core #1 core #3 core #2 node #1 core #0 core #1 core #3 core #2

Slide 22

Slide 22 text

INSERT DESIGNATOR, IF NEEDED 22 Guest vCPU Placement node #0 core #0 core #1 core #3 core #2 node #1 core #0 core #1 core #3 core #2

Slide 23

Slide 23 text

INSERT DESIGNATOR, IF NEEDED 23 Guest vCPU Placement node #0 core #0 core #1 core #3 core #2 node #1 core #0 core #1 core #3 core #2

Slide 24

Slide 24 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 test.pinned $ openstack flavor set test.pinned \ --property hw:cpu_policy=dedicated 24 Guest vCPU Placement

Slide 25

Slide 25 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 6 --ram 4096 --disk 20 test.pinned $ openstack flavor set test.pinned \ --property hw:cpu_policy=dedicated 25 Guest vCPU Placement

Slide 26

Slide 26 text

INSERT DESIGNATOR, IF NEEDED 26 Guest vCPU Placement node #0 core #0 core #1 core #3 core #2 node #1 core #0 core #1 core #3 core #2

Slide 27

Slide 27 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 6 --ram 4096 --disk 20 test.pinned $ openstack flavor set test.pinned \ --property hw:cpu_policy=dedicated 27 Guest vCPU Placement

Slide 28

Slide 28 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 6 --ram 4096 --disk 20 test.pinned $ openstack flavor set test.pinned \ --property hw:cpu_policy=dedicated --property hw:numa_nodes=2 28 Guest vCPU Placement

Slide 29

Slide 29 text

INSERT DESIGNATOR, IF NEEDED 29 Guest vCPU Placement node #0 core #0 core #1 core #3 core #2 node #1 core #0 core #1 core #3 core #2

Slide 30

Slide 30 text

INSERT DESIGNATOR, IF NEEDED 30 NUMA in OpenStack ● NUMA Guest Topologies ● Guest vCPU Placement ● PCI NUMA Affinity ● vGPU, Neutron Network NUMA Affinity

Slide 31

Slide 31 text

INSERT DESIGNATOR, IF NEEDED 31 NUMA in OpenStack ● NUMA Guest Topologies ● Guest vCPU Placement ● PCI NUMA Affinity ● vGPU, Neutron Network NUMA Affinity

Slide 32

Slide 32 text

INSERT DESIGNATOR, IF NEEDED 32

Slide 33

Slide 33 text

INSERT DESIGNATOR, IF NEEDED 33

Slide 34

Slide 34 text

INSERT DESIGNATOR, IF NEEDED 34 PCI NUMA Affinity node #0 core #1 core #2 core #5 core #4 core #0 core #3 node #1 core #1 core #2 core #5 core #4 core #0 core #3

Slide 35

Slide 35 text

INSERT DESIGNATOR, IF NEEDED [pci] alias = '{ "name": "QuickAssist", "product_id": "0443", "vendor_id": "8086", "device_type": "type-PCI" }' 35 PCI NUMA Affinity

Slide 36

Slide 36 text

INSERT DESIGNATOR, IF NEEDED $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 test.pci $ openstack flavor set test.pci \ --property pci_passthrough:alias=QuickAssist:1 36 PCI NUMA Affinity

Slide 37

Slide 37 text

INSERT DESIGNATOR, IF NEEDED 37

Slide 38

Slide 38 text

INSERT DESIGNATOR, IF NEEDED 38 PCI NUMA Affinity node #0 core #1 core #2 core #5 core #4 core #0 core #3 node #1 core #1 core #2 core #5 core #4 core #0 core #3

Slide 39

Slide 39 text

INSERT DESIGNATOR, IF NEEDED 39 PCI NUMA Affinity node #0 core #1 core #2 core #5 core #4 core #0 core #3 node #1 core #1 core #2 core #5 core #4 core #0 core #3

Slide 40

Slide 40 text

INSERT DESIGNATOR, IF NEEDED [pci] alias = '{ "name": "QuickAssist", "product_id": "0443", "vendor_id": "8086", "device_type": "type-PCI" }' 40 PCI NUMA Affinity

Slide 41

Slide 41 text

INSERT DESIGNATOR, IF NEEDED [pci] alias = '{ "name": "QuickAssist", "product_id": "0443", "vendor_id": "8086", "device_type": "type-PCI", "numa_policy": "preferred" # or 'legacy' or 'required' }' 41 PCI NUMA Affinity

Slide 42

Slide 42 text

INSERT DESIGNATOR, IF NEEDED 42 PCI NUMA Affinity node #0 core #1 core #2 core #5 core #4 core #0 core #3 node #1 core #1 core #2 core #5 core #4 core #0 core #3

Slide 43

Slide 43 text

INSERT DESIGNATOR, IF NEEDED 43 NUMA in OpenStack ● NUMA Guest Topologies ● Guest vCPU Placement ● PCI NUMA Affinity ● vGPU, Neutron Network NUMA Affinity

Slide 44

Slide 44 text

INSERT DESIGNATOR, IF NEEDED 44 NUMA in OpenStack ● NUMA Guest Topologies ● Guest vCPU Placement ● PCI NUMA Affinity ● vGPU, Neutron Network NUMA Affinity

Slide 45

Slide 45 text

INSERT DESIGNATOR, IF NEEDED 45 NUMA in OpenStack ● NUMA Guest Topologies ● Guest vCPU Placement ● PCI NUMA Affinity ● vGPU, Neutron Network NUMA Affinity *coming soon*

Slide 46

Slide 46 text

Common Questions

Slide 47

Slide 47 text

INSERT DESIGNATOR, IF NEEDED 47 Common Questions ● Can I choose what host NUMA nodes my guest runs on?

Slide 48

Slide 48 text

INSERT DESIGNATOR, IF NEEDED 48 Common Questions ● Can I choose what host NUMA nodes my guest runs on? ○ We don’t support this by design

Slide 49

Slide 49 text

INSERT DESIGNATOR, IF NEEDED 49 Common Questions ● Can I choose what host NUMA nodes my guest runs on? ○ We don’t support this by design ● Why would I want a multi-node guest?

Slide 50

Slide 50 text

INSERT DESIGNATOR, IF NEEDED 50 Common Questions ● Can I choose what host NUMA nodes my guest runs on? ○ We don’t support this by design ● Why would I want a multi-node guest? ○ By necessity ■ Large core counts ■ Multiple PCI devices with different NUMA affinities ○ Application requirements

Slide 51

Slide 51 text

INSERT DESIGNATOR, IF NEEDED 51 Common Questions ● Can I choose what host NUMA nodes my guest runs on? ○ We don’t support this by design ● Why would I want a multi-node guest? ○ By necessity ■ Large core counts ■ Multiple PCI devices with different NUMA affinities ○ Application requirements ● Can a guest’s NUMA nodes share the same host node?

Slide 52

Slide 52 text

INSERT DESIGNATOR, IF NEEDED 52 Common Questions ● Can I choose what host NUMA nodes my guest runs on? ○ We don’t support this by design ● Why would I want a multi-node guest? ○ By necessity ■ Large core counts ■ Multiple PCI devices with different NUMA affinities ○ Application requirements ● Can a guest’s NUMA nodes share the same host node? ○ Not at the moment

Slide 53

Slide 53 text

INSERT DESIGNATOR, IF NEEDED 53 Common Misconceptions

Slide 54

Slide 54 text

INSERT DESIGNATOR, IF NEEDED 54 Common Misconceptions ● Host NUMA node selection ○ You can’t dictate what node is used - nova must decide

Slide 55

Slide 55 text

INSERT DESIGNATOR, IF NEEDED 55 Common Misconceptions ● Host NUMA node selection ○ You can’t dictate what node is used - nova must decide ● Host sockets != NUMA nodes ○ Cluster-on-Die is a thing

Slide 56

Slide 56 text

INSERT DESIGNATOR, IF NEEDED 56 Common Misconceptions ● Host NUMA node selection ○ You can’t dictate what node is used - nova must decide ● Host sockets != NUMA nodes ○ Cluster-on-Die is a thing ● Guest sockets != NUMA nodes ○ You can specify hw:numa_nodes and hw:cpu_sockets

Slide 57

Slide 57 text

INSERT DESIGNATOR, IF NEEDED 57 Common Misconceptions ● Host NUMA node selection ○ You can’t dictate what node is used - nova must decide ● Host sockets != NUMA nodes ○ Cluster-on-Die is a thing ● Guest sockets != NUMA nodes ○ You can specify hw:numa_nodes and hw:cpu_sockets ● CPU pinning isn’t a requirement ○ It’s just common in these scenarios

Slide 58

Slide 58 text

Questions?

Slide 59

Slide 59 text

THANK YOU plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews

Slide 60

Slide 60 text

INSERT DESIGNATOR, IF NEEDED 60 Resources You might want to know about these... ● RHEL NUMA Tuning Guide ● Attaching physical PCI devices to guests ● Nova Flavors Guide