GPU on OpenStack日本語版

Transcript

1. GPU on OpenStack ͓͓ͨɹ·͞;Έ @masafumiohta

2. ͓લͩΕʁ> ๭SIerۈ຿ ͍͔ͭ͘ͷOpenStack PoCʹࢀՃɺٴͼΞυόΠε OpenStackͷఏҊɾޚ૬ஊ΋ঝͬͯ·͢ɻ OpenStackίʔυಡΜͰमਖ਼ͨ͠Γͱ͔… ଓ͖͸WebͰ https://jp.linkedin.com/in/ohtamasafumi

3. ͸͡Ίʹ OpenStack͸ಛघར༻΁ Sahara(HadoopͷOpenStack)ͳͲͳͲ ΄ͱΜͲ͕·ͱΊΒΕͯͳ͙ͬͯ͘ௐ΂Δ͔͠ͳ͍ ͍ΘΏΔʰυΩϡϝϯτϩετʱ docs.openstack.orgɹʹ͋͛ͯ΄͍͠ɻ

4. GPU on OpenStackͱ͸

5. GPUτϨϯυ GPU͸ϝχʔίΞ໨త ֤MPUίΞ͸Ұݸ͋ͨΓ͸খ͘͞஗͍͕ز͔ͭͷܭࢉ (MATLABܥͳͲ)Ͱ͸༗ޮ GPUͷμΠूੵʹର͢Δলిྗ͸HPCϢʔβʹ ͱͬͯ͸େ͖͍ɻ ίϯύΫτͳγεςϜ͸লిྗɾলεϖʔε͕ඞਢͳ೔ຊ ͷHPCγεςϜʹ͸େ͖͍

6. GPU͸Ͳ͏΍ͬͯ࢖͏ͷ͔ʁ LinuxͷPCIύεεϧʔٕज़Ͱ࢖͏ AWS͸ଟ෼͜ͷϕʔε KVMʹґଘ VSphere͸GPUίΞͷεϓϦοτ͕Մೳ VSphere΍XenͷΑ͏ͳGPUεϓϦοτ͸Մೳ͔ʁ ݱߦ͸Ͱ͖ͳ͍͕Nvidia͕αϙʔτ༧ఆ

7. GPU Docker DockerΛ͔ͭͬͨGPUͷར༻ ίΞ෼ׂ͸Ͱ͖ͳ͍͕λεΫʹԠͯ͡GPUίΞར༻Λ ࣗಈར༻෼ׂɻ ࠷ۙͷGPU on OpenStackͱ͍͑͹͜ͷGPU Dockerར ༻

   ͨͩɺύεεϧʔ+KVMΑΓ͸ύϑΥʔϚϯε͸Ͱͳ͍

8. GPU OpenStack͸୭ͷͨΊ HPCͷςϯϙϥϦར༻ ͪΐͬͱܭࢉ΍ͬͯऴΘͬͨΒVM͝ͱյ͢ ͍͔ͭ͘ͷVMΛ͔ͭͬͯ؆୯ʹHPCάϦουͱ͔.. EC2ͷΑ͏ʹHPCΛ࢖ͬͨΓ͢Δ ಺෦ར༻Ͱ࢖͏ɺಛʹ੡଄ۀͰEC2ͳͲΫϥ΢υʹग़ͪ͠Ό ͍͚ͳ͍ਓ޲͚

9. GPU on OpenStackΛ࢖͏

10. PCIύεεϧʔ(1) PCIσόΠεΛμΠϨΫτʹVMʹͭͳ͙ ෺ཧϗετΑΓ͔ΒPCIσόΠεΛ੾Γ཭͢ඞཁ͕͋Δɻ KVMͷػೳʹࠨӈ͞ΕΔ͜ͱʹͳ͍ͬͯΔɻ OpenStackͱ͸ؔ܎ͳ͍ ҰͭͷσόΠεʹҰͭͷVM GPUࣗମ͸֤VMʹର͠ڞ༗͓Αͼ෼ׂ͸ෆՄೳ ͋͘·ͰKVMͷ੍ݶɺOpenStackͰ͸ͳ͍ɻ

11. PCIύεεϧʔ(2) RedhatͰ͸ެࣜαϙʔτ ͔͠͠ར༻͸ੵۃਪ঑͍ͯ͠ͳ͍ʢ͋΍͍͠ʁ) ࣮ࡍRedhatͰ΋ϋϚΔ.. UbuntuͰ͸υΩϡϝϯτԽ͞Ε͍ͯͳ͍ άάͬͯ৘ใΛ͕͞͞ͳ͖Ό͍͚ͳ͍ɻ

12. K Linux OS for KVM hypervisor GPU Driver App Instance

    VMM/KVM IOMMU/Vt-d PCI Express x16 Linux/Win OS ComputeNode GPU Card GPU Card Nova Compute Nova Scheduler AMQP Nova API Linux OS ControllerNode ਤ:GPUύεεϧʔ͸Ͳ͏΍ͬͯOpenStackͰಈ͔͘ʢ໼ҹ͕Πϯελϯεൃߦϓϩηεʣ

13. KVMϗετͷGPU ϗετͷGPUΛνΣοΫ͢Δ: lspci -nn | grep -i nvidia lspci -nn

    | grep -i nvidia 88:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:11b4] (rev a1) 88:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) GPUͷϢχοτ͝ͱύεεϧʔ͢Δඞཁ͕͋Δ GPUϗετ͚ͩͰͳ͘HDMIͳͲϢχοτ͝ͱ੾Γ཭͢ඞཁ͋Γ Ͱͳ͍ͱVM্Ͱಈ͍ͯ͘Εͳ͍=ύεεϧʔͰ͖ͳ͍

14. GPUͷϙʔτΛΈΔ GPUʹ͸HDMIϏσΦͱΦʔσΟΦ͕͋ΔͷͰ ͜Ε͝ͱύεεϧʔ͢Δඞཁ͕༗Δ

15. IOMMUͷηοτΞοϓ ෺ཧσόΠεΛ࢖͏ͨΊͷԾ૝γεςϜʹ͓͍ͯ IOMMU(Input/Output Memory Management Unit) ηοτΞο ϓ͸ඞਢ ΋ͪΖΜ intel

    vt-d (I/OԾ૝Խ)͸Φϯʹ (EFI/BIOS΋νΣοΫ) grubͰͷηοτΞοϓ΋ඞཁ:/etc/default/grubΛมߋهड़ GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1”

16. pci-stub pci-stub ͸෺ཧPCIσόΠεΛϗετͰར༻Ͱ͖ͳ͍Α͏ʹ͢Δ ͨΊʹઃఆ͢Δ /etc/moduleʹσϑΥϧτͰઃఆ͞Ε͍ͯͳ͍ͷͰઃఆ͢Δɻؔ ࿈͢Δίϯϙʔωϯτ΋ઃఆ͢Δ (vfio,kvm)ɻ pci_stub vfio vfio_iommu_type1

    vfio_pci kvm kvm_intel

17. VFIO(1) ύεεϧʔ͞ΕΔ΋ͷΛ෺ཧσόΠε͔Β੾Γ཭ͨ͢ ΊʹVFIO(Virtual Function IO)্ʹ௥Ճ͢Δඞཁ͕͋Δɻ ͜ΕΒͷσόΠεΛramfsͰೝࣝ͞ΕΔͷΛ๷͙ /etc/initramfs-tools/modules to initramfs (ubuntu)

    echo ‘pci_stub ids=10de:11b4,10de:0e0a’ >> /etc/initramfs-tools/modules sudo update-initramfs -u && sudo reboot

18. VFIO(2) ͜ΕΒͷσόΠε͸·ͨbootͷࡍʹ΋ೝࣝ͞Εͳ͍ Α͏ʹ͢Δɻ͢΂ͯͷboot upγʔέϯεͰ෺ཧσό ΠεͰͷೝࣝΛ཈ࢭ͢Δɻ /etc/modprobe.d/blacklist.conf ʹ௥Ճ: blacklist nvidia blacklist

    nvidia-uvm Nvidiaޓ׵υϥΠό΋ೝࣝͤ͞ͳ͍Α͏ʹ௥Ճ blacklist nouveau

19. ෺ཧ͔Β੾Γ཭͢ pci-stubΛ͔ͭͬͯ෺ཧσόΠεΑΓ੾Γ཭͠VMͱ͚ͬͭ͘Δɻσ όΠεIDΛnew_idʹ௥Ճ͢Δɻ·ͨؔ࿈ࣝผࢠΛ੾Γ཭͠ɺVMͱ ͚ͬͭ͘Δɻ echo 11de 11b4 > /sys/bus/pci/drivers/pci-stub/new_id echo

    11de 0e0a > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:88:00.0 > /sys/bus/pci/devices/0000:88:00.0/driver/unbind echo 0000:88:00.1 > /sys/bus/pci/devices/0000:88:00.1/driver/unbind echo 0000:88:00.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:88:00.1 > /sys/bus/pci/drivers/pci-stub/bind ෺ཧϚγϯ͔Β੾Γ཭͞Ε͔ͨɺ’claimed’ʹͳ͍ͬͯΔ͔ௐ΂Δɻ pci-stub 0000:88:00.1: claimed by stub

20. modprobe /etc/modprobe.d/blacklist.conf pci-stab /sys/bus/pci/drivers/pci-stub/ /sys/bus/pci/devices/$(Identifier)/driver/unbind ramfs /etc/initramfs-tools/modules GRUB /etc/default/grub modules

    /etc/modules UEFI/BIOS Vt-d ਤ:GPUύεεϧʔͰComputeNodeͷBOOTϓϩηε্Ͱઃఆ͢Δ΋ͷ B O O T ϓ ϩ η ε IOMMU IOMMU BLACK LIST BLACK LIST IOMMU BLACK LIST

21. ෺ཧσόΠε pci-stub (Ծ૝Ͱ࢖͏) GPUϢχοτ (σόΠεશମ) GPUΛ෺ཧσόΠε͔Β੾Γ཭͠ɺ Ծ૝σόΠεʹ෇͚ସ͑Δɻ echo 11de 11b4

    > /sys/bus/pci/drivers/pci-stub/new_id echo 11de 0e0a > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:88:00.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:88:00.1 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:88:00.0 > /sys/bus/pci/devices/0000:88:00.0/driver/unbind echo 0000:88:00.1 > /sys/bus/pci/devices/0000:88:00.1/driver/unbind

22. ͞ΒʹGPUΛՃ͑Δ(1) lspciͷ݁ՌΛ֬ೝɺ2ͭͷσόΠεID͕݁Ռͱͯ͠ݟ͑Δɺ ͜ͷݟ͑ํ͸γεςϜʹґଘɻ lspci -nn | grep -i nvidia 88:00.0

    VGA compatible controller [0300]: NVIDIA Corporation Device [10de:11b4] (rev a1) 88:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 84:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:11b4] (rev a1) 84:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

23. ͞ΒʹGPUΛՃ͑Δ(2) pci-stubΛར༻͠͞ΒʹσόΠεΛύεεϧʔ͢Δɻ echo 0000:84:00.0 > /sys/bus/pci/devices/0000:84:00.0/driver/unbind echo 0000:84:00.1 > /sys/bus/pci/devices/0000:84:00.1/driver/unbind

    echo 0000:84:00.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:84:00.1 > /sys/bus/pci/drivers/pci-stub/bind CUDAΞϓϦΛ࢖͏্ͰಉػछͷGPUͷ௥Ճ͕ඞਢ(͕ͪ ͏΋ͷ͸ෆՄ)ɺಉ͡΋ͷͰ͋Δ͔ฉ͔ΕΔ͜ͱ͕͋Δɻ /nbody -benchmark -numdevices=2 -num bodies=65536

24. ͞ΒʹGPUΛՃ͑Δ(3) ύεεϧʔʹ੒ޭ͢ΔͱVM্Ͱಈ͍͍ͯΔͷ͕lspciʹͯ νΣοΫͰ͖Δɻ ubuntu@guestos$ lspci -nn | grep -i nvidia

    00:07.0 VGA compatible controller [0300]: NVIDIA Corporation GK104GL [Quadro K4200] [10de:11b4] (rev a1) 00:08.0 VGA compatible controller [0300]: NVIDIA Corporation GK104GL [Quadro K4200] [10de:11b4] (rev a1)

25. Nova΁ͷ௥ه(1) ComputeNodeͷwhitelist͕PCIύεεϧʔͱGPUͷ VM΁ͷσϓϩΠϝϯτͰඞཁɻ /etc/nova/nova.conf ʹ pci_passthrough_whitelist Λ௥ه pci_passthrough_whitelist={"name":"K4200","vendor_id":"10de","product_id": "11b4"}

26. Nova΁ͷ௥ه(2) ίϯτϩʔϥʔϊʔυͷnova aliasͷઃఆ͕ඞཁɺޙ ड़͢Δflavor-key Ͱར༻͢Δɻ to /etc/nova/nova.confʹ pci_aliasesΛ௥ه pci_alias={“name”:”K4200”,"vendor_id":"10de","product_id":"11b4"}

27. Nova΁ͷ௥ه(3) ίϯτϩʔϥϊʔυʹͯPCIύεεϧʔϑΟϧλʔΛ novaʹ௥Ճɻ /etc/nova/nova.conf ʹΞϯμʔϥΠϯ෦෼௥Ճ scheduler_available_filters=nova.scheduler.filters.all_filters scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciP assthroughFilter scheduler_default_filters=DifferentHostFilter,RetryFilter,AvailabilityZoneFilter, RamFilter,CoreFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,Imag

    ePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,Aggre gateInstanceExtraSpecsFilter,PciPassthroughFilter

28. nova alias GPUΛ࢖͏ͨΊʹflavor-keyΛ௥ه͢ΔɻϑϨʔόʹset໊ Ͱطଘflavorʹର͠pciύεεϧʔͷͰܾΊͨGPUͷalias ໊ɺ࢖͍͍ͨGPUϢχοτ਺Λ௥Ճ͢Δɻ ্ه͸લड़·ͰͷաఔͰɺQuadro K4200Λ2Ϣχοτ௥Ճ͍ͯ͠ Δ͜ͱ͕લఏͳͷͰɺҎԼͷΑ͏ͳهड़ʹͳΔɻ pci_aliasΛK4200ͱ͍ͯ͠ΔͨΊҎԼͷهड़ͱͳΔɻ nova

    flavor-key $flavor_name set “pci_passthrough:alias”=“K4200:$amount_of_gpu”

29. ط஌ͷ໰୊

30. Ϋϥ΢υΠϝʔδͷ໰୊ Πϝʔδ͕GPUΛ࢖͏ʹ͸ͱͯ΋খ͘͞qemu-imgͰΠϝʔδͷϦα ΠζΛͯ͠΍Δඞཁ͕༗Δɻ CUDA driverΛΠϯετʔϧ͢Δʹ͸perl-packagesΛطଘͷΫϥ΢υ Πϝʔδʹ͸͍Εͯ΍Δඞཁ͕͋Δɻ CUDAυϥΠό͸.deb or .rpMύοέʔδͰ΋όΠφϦϑΝΠϧͰͳ͘ɺΠϯε τʔϧ࣌ʹιʔεΛmakeͰΠϯετʔϧ͢ΔܗΛͱ͍ͬͯΔ=.runϑΝΠϧͱม

    ΘΒͣɻ Nvidia͸কདྷͷϦϦʔεͰspecϑΝΠϧʹperlؔ࿈ϑΝΠϧͷ௥هΛ͢ΔܗͰfix ͢Δ༧ఆɻ 7.6Ҏ߱Ͱగਖ਼ͯ͘͠ΕΔɺͱ͍͍͕ͬͯͨ…(·ͩςετͯ͠ͳ͍ɻʣ

31. Windows as VDI Windows্ͷCUDA͸ͪΌΜͱΠϯετʔϧͰ͖Ε͹ૣ ͘ͳΔ͚Ͳ࣌ͨ·ΧΫΧΫͯ͠͠·͏ɻ vmͷdisk speedͷ໰୊ɺΤϑΣϝϥϧʹ͢ΔͳΓૣ͍σΟεΫΛ ࢖͏ͳΓͰͦͦ͜͜ղܾ͢Δ(SSD,NVMe or..etc) vm͸context

    switchͰr/wΛߦ͍ͬͯΔͷͰϔϏʔϫʔΫϩʔυ ͷCUDAͳΓ͸͜ͷΑ͏ͳΧΫΧΫঢ়ଶΛى͜͢Մೳੑ͕͋Δɻ ͜ͷ͋ͱ͸ޙ೚ʹ·͔͕ͤͨɺগ͠͸ΤϑΣϝϥϧͰվળͨ͠Β͍͠ ސ٬͸·ɺͳΜͱ͔ͳͬͨͱͱΓ͋͑ͣͷຬ଍͸΋Βͬͨɻ

32. ϥΠϒϚΠάϨʔγϣϯ ϥΠϒϚΠΫϨʔγϣϯ͸Ͱ͖ͳ͍ɺvm͕ݹ͍ϗετͷ઀ଓ৘ ใΛ੾Γ཭ͤͳ͍ɻݹ͍ϗετͷ৘ใΛѲͬͨ··ʹͳΔɻ ϫʔΫΞϥ΢ϯυ:nova.pci_devicesͷMySQLͷDB৘ใΛফ͠ ͯɺݹ͍ϗετΛ࠶ىಈ͢Δʼҙຯͳ͠ʂ | 2016-08-11 00:54:45 | 2016-08-19

    04:58:01 | NULL | 0 | 45 | 21 | 0000:84:00.0 | 11b4 | 10de | type-PCI | pci_0000_84_00_0 | label_10de_11b4 | available | {} | NULL | NULL | 1 | <<-- old-host | 2016-08-11 00:54:45 | 2016-08-19 04:58:01 | NULL | 0 | 48 | 21 | 0000:88:00.0 | 11b4 | 10de | type-PCI | pci_0000_88_00_0 | label_10de_11b4 | available | {} | NULL | NULL | 1 | <<-- old-host

33. ·ͱΊ OpenStackͷಛघར༻͕Ͱ͖Δ͔Ͳ͏͔͸౔୆ͷOS࣍ୈ Ͱ͢ɻ ౔୆ͷOS͕μϝͳͷʹOpenStack͕..ͱ͍͏ͷ͸Πέͯͳ͍ਓ͕͍ ͏͓ݴ༿… αʔόܥOS͸ҙ֎ʹ͍Ζ͍ΖΠέͯ·ͤΜ…໰୊͋Γ͋ΓͰ͢ɻ GPUͷར༻͸ࠓ͸͜ͷύεεϧʔ͕ݱߦͰ͕͢ɺNvidia ΍Intel࣍ୈ͔ͱ(AMD͸…Ͳ͏͢ΔΜͩΖ͏…)

34. Thank you Masafumi Ohta @masafumiohta