GPU on OpenStack日本語版

日本語バージョン作りました。若干訂正も加えました。

Transcript

1. GPU on OpenStack ͓͓ͨɹ·͞;Έ @masafumiohta
2. ͓લͩΕʁ> ๭SIerۈ຿ ͍͔ͭ͘ͷOpenStack PoCʹࢀՃɺٴͼΞυόΠε OpenStackͷఏҊɾޚ૬ஊ΋ঝͬͯ·͢ɻ OpenStackίʔυಡΜͰमਖ਼ͨ͠Γͱ͔… ଓ͖͸WebͰ https://jp.linkedin.com/in/ohtamasafumi
3. ͸͡Ίʹ OpenStack͸ಛघར༻΁ Sahara(HadoopͷOpenStack)ͳͲͳͲ ΄ͱΜͲ͕·ͱΊΒΕͯͳ͙ͬͯ͘ௐ΂Δ͔͠ͳ͍ ͍ΘΏΔʰυΩϡϝϯτϩετʱ docs.openstack.orgɹʹ͋͛ͯ΄͍͠ɻ
4. GPU on OpenStackͱ͸
5. GPUτϨϯυ GPU͸ϝχʔίΞ໨త ֤MPUίΞ͸Ұݸ͋ͨΓ͸খ͘͞஗͍͕ز͔ͭͷܭࢉ (MATLABܥͳͲ)Ͱ͸༗ޮ GPUͷμΠूੵʹର͢Δলిྗ͸HPCϢʔβʹ ͱͬͯ͸େ͖͍ɻ ίϯύΫτͳγεςϜ͸লిྗɾলεϖʔε͕ඞਢͳ೔ຊ ͷHPCγεςϜʹ͸େ͖͍
6. GPU͸Ͳ͏΍ͬͯ࢖͏ͷ͔ʁ LinuxͷPCIύεεϧʔٕज़Ͱ࢖͏ AWS͸ଟ෼͜ͷϕʔε KVMʹґଘ VSphere͸GPUίΞͷεϓϦοτ͕Մೳ VSphere΍XenͷΑ͏ͳGPUεϓϦοτ͸Մೳ͔ʁ ݱߦ͸Ͱ͖ͳ͍͕Nvidia͕αϙʔτ༧ఆ
7. GPU Docker DockerΛ͔ͭͬͨGPUͷར༻ ίΞ෼ׂ͸Ͱ͖ͳ͍͕λεΫʹԠͯ͡GPUίΞར༻Λ ࣗಈར༻෼ׂɻ ࠷ۙͷGPU on OpenStackͱ͍͑͹͜ͷGPU Dockerར ༻ ͨͩɺύεεϧʔ+KVMΑΓ͸ύϑΥʔϚϯε͸Ͱͳ͍
8. GPU OpenStack͸୭ͷͨΊ HPCͷςϯϙϥϦར༻ ͪΐͬͱܭࢉ΍ͬͯऴΘͬͨΒVM͝ͱյ͢ ͍͔ͭ͘ͷVMΛ͔ͭͬͯ؆୯ʹHPCάϦουͱ͔.. EC2ͷΑ͏ʹHPCΛ࢖ͬͨΓ͢Δ ಺෦ར༻Ͱ࢖͏ɺಛʹ੡଄ۀͰEC2ͳͲΫϥ΢υʹग़ͪ͠Ό ͍͚ͳ͍ਓ޲͚
9. GPU on OpenStackΛ࢖͏
10. PCIύεεϧʔ(1) PCIσόΠεΛμΠϨΫτʹVMʹͭͳ͙ ෺ཧϗετΑΓ͔ΒPCIσόΠεΛ੾Γ཭͢ඞཁ͕͋Δɻ KVMͷػೳʹࠨӈ͞ΕΔ͜ͱʹͳ͍ͬͯΔɻ OpenStackͱ͸ؔ܎ͳ͍ ҰͭͷσόΠεʹҰͭͷVM GPUࣗମ͸֤VMʹର͠ڞ༗͓Αͼ෼ׂ͸ෆՄೳ ͋͘·ͰKVMͷ੍ݶɺOpenStackͰ͸ͳ͍ɻ
11. PCIύεεϧʔ(2) RedhatͰ͸ެࣜαϙʔτ ͔͠͠ར༻͸ੵۃਪ঑͍ͯ͠ͳ͍ʢ͋΍͍͠ʁ) ࣮ࡍRedhatͰ΋ϋϚΔ.. UbuntuͰ͸υΩϡϝϯτԽ͞Ε͍ͯͳ͍ άάͬͯ৘ใΛ͕͞͞ͳ͖Ό͍͚ͳ͍ɻ
12. K Linux OS for KVM hypervisor GPU Driver App Instance VMM/KVM IOMMU/Vt-d PCI Express x16 Linux/Win OS ComputeNode GPU Card GPU Card Nova Compute Nova Scheduler AMQP Nova API Linux OS ControllerNode ਤ:GPUύεεϧʔ͸Ͳ͏΍ͬͯOpenStackͰಈ͔͘ʢ໼ҹ͕Πϯελϯεൃߦϓϩηεʣ
13. KVMϗετͷGPU ϗετͷGPUΛνΣοΫ͢Δ: lspci -nn | grep -i nvidia lspci -nn | grep -i nvidia 88:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:11b4] (rev a1) 88:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) GPUͷϢχοτ͝ͱύεεϧʔ͢Δඞཁ͕͋Δ GPUϗετ͚ͩͰͳ͘HDMIͳͲϢχοτ͝ͱ੾Γ཭͢ඞཁ͋Γ Ͱͳ͍ͱVM্Ͱಈ͍ͯ͘Εͳ͍=ύεεϧʔͰ͖ͳ͍
14. GPUͷϙʔτΛΈΔ GPUʹ͸HDMIϏσΦͱΦʔσΟΦ͕͋ΔͷͰ ͜Ε͝ͱύεεϧʔ͢Δඞཁ͕༗Δ
15. IOMMUͷηοτΞοϓ ෺ཧσόΠεΛ࢖͏ͨΊͷԾ૝γεςϜʹ͓͍ͯ IOMMU(Input/Output Memory Management Unit) ηοτΞο ϓ͸ඞਢ ΋ͪΖΜ intel vt-d (I/OԾ૝Խ)͸Φϯʹ (EFI/BIOS΋νΣοΫ) grubͰͷηοτΞοϓ΋ඞཁ:/etc/default/grubΛมߋهड़ GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1”
16. pci-stub pci-stub ͸෺ཧPCIσόΠεΛϗετͰར༻Ͱ͖ͳ͍Α͏ʹ͢Δ ͨΊʹઃఆ͢Δ /etc/moduleʹσϑΥϧτͰઃఆ͞Ε͍ͯͳ͍ͷͰઃఆ͢Δɻؔ ࿈͢Δίϯϙʔωϯτ΋ઃఆ͢Δ (vfio,kvm)ɻ pci_stub vfio vfio_iommu_type1 vfio_pci kvm kvm_intel
17. VFIO(1) ύεεϧʔ͞ΕΔ΋ͷΛ෺ཧσόΠε͔Β੾Γ཭ͨ͢ ΊʹVFIO(Virtual Function IO)্ʹ௥Ճ͢Δඞཁ͕͋Δɻ ͜ΕΒͷσόΠεΛramfsͰೝࣝ͞ΕΔͷΛ๷͙ /etc/initramfs-tools/modules to initramfs (ubuntu) echo ‘pci_stub ids=10de:11b4,10de:0e0a’ >> /etc/initramfs-tools/modules sudo update-initramfs -u && sudo reboot
18. VFIO(2) ͜ΕΒͷσόΠε͸·ͨbootͷࡍʹ΋ೝࣝ͞Εͳ͍ Α͏ʹ͢Δɻ͢΂ͯͷboot upγʔέϯεͰ෺ཧσό ΠεͰͷೝࣝΛ཈ࢭ͢Δɻ /etc/modprobe.d/blacklist.conf ʹ௥Ճ: blacklist nvidia blacklist nvidia-uvm Nvidiaޓ׵υϥΠό΋ೝࣝͤ͞ͳ͍Α͏ʹ௥Ճ blacklist nouveau
19. ෺ཧ͔Β੾Γ཭͢ pci-stubΛ͔ͭͬͯ෺ཧσόΠεΑΓ੾Γ཭͠VMͱ͚ͬͭ͘Δɻσ όΠεIDΛnew_idʹ௥Ճ͢Δɻ·ͨؔ࿈ࣝผࢠΛ੾Γ཭͠ɺVMͱ ͚ͬͭ͘Δɻ echo 11de 11b4 > /sys/bus/pci/drivers/pci-stub/new_id echo 11de 0e0a > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:88:00.0 > /sys/bus/pci/devices/0000:88:00.0/driver/unbind echo 0000:88:00.1 > /sys/bus/pci/devices/0000:88:00.1/driver/unbind echo 0000:88:00.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:88:00.1 > /sys/bus/pci/drivers/pci-stub/bind ෺ཧϚγϯ͔Β੾Γ཭͞Ε͔ͨɺ’claimed’ʹͳ͍ͬͯΔ͔ௐ΂Δɻ pci-stub 0000:88:00.1: claimed by stub
20. modprobe /etc/modprobe.d/blacklist.conf pci-stab /sys/bus/pci/drivers/pci-stub/ /sys/bus/pci/devices/$(Identifier)/driver/unbind ramfs /etc/initramfs-tools/modules GRUB /etc/default/grub modules /etc/modules UEFI/BIOS Vt-d ਤ:GPUύεεϧʔͰComputeNodeͷBOOTϓϩηε্Ͱઃఆ͢Δ΋ͷ B O O T ϓ ϩ η ε IOMMU IOMMU BLACK LIST BLACK LIST IOMMU BLACK LIST
21. ෺ཧσόΠε pci-stub (Ծ૝Ͱ࢖͏) GPUϢχοτ (σόΠεશମ) GPUΛ෺ཧσόΠε͔Β੾Γ཭͠ɺ Ծ૝σόΠεʹ෇͚ସ͑Δɻ echo 11de 11b4 > /sys/bus/pci/drivers/pci-stub/new_id echo 11de 0e0a > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:88:00.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:88:00.1 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:88:00.0 > /sys/bus/pci/devices/0000:88:00.0/driver/unbind echo 0000:88:00.1 > /sys/bus/pci/devices/0000:88:00.1/driver/unbind
22. ͞ΒʹGPUΛՃ͑Δ(1) lspciͷ݁ՌΛ֬ೝɺ2ͭͷσόΠεID͕݁Ռͱͯ͠ݟ͑Δɺ ͜ͷݟ͑ํ͸γεςϜʹґଘɻ lspci -nn | grep -i nvidia 88:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:11b4] (rev a1) 88:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1) 84:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:11b4] (rev a1) 84:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
23. ͞ΒʹGPUΛՃ͑Δ(2) pci-stubΛར༻͠͞ΒʹσόΠεΛύεεϧʔ͢Δɻ echo 0000:84:00.0 > /sys/bus/pci/devices/0000:84:00.0/driver/unbind echo 0000:84:00.1 > /sys/bus/pci/devices/0000:84:00.1/driver/unbind echo 0000:84:00.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:84:00.1 > /sys/bus/pci/drivers/pci-stub/bind CUDAΞϓϦΛ࢖͏্ͰಉػछͷGPUͷ௥Ճ͕ඞਢ(͕ͪ ͏΋ͷ͸ෆՄ)ɺಉ͡΋ͷͰ͋Δ͔ฉ͔ΕΔ͜ͱ͕͋Δɻ /nbody -benchmark -numdevices=2 -num bodies=65536
24. ͞ΒʹGPUΛՃ͑Δ(3) ύεεϧʔʹ੒ޭ͢ΔͱVM্Ͱಈ͍͍ͯΔͷ͕lspciʹͯ νΣοΫͰ͖Δɻ ubuntu@guestos$ lspci -nn | grep -i nvidia 00:07.0 VGA compatible controller [0300]: NVIDIA Corporation GK104GL [Quadro K4200] [10de:11b4] (rev a1) 00:08.0 VGA compatible controller [0300]: NVIDIA Corporation GK104GL [Quadro K4200] [10de:11b4] (rev a1)
25. Nova΁ͷ௥ه(1) ComputeNodeͷwhitelist͕PCIύεεϧʔͱGPUͷ VM΁ͷσϓϩΠϝϯτͰඞཁɻ /etc/nova/nova.conf ʹ pci_passthrough_whitelist Λ௥ه pci_passthrough_whitelist={"name":"K4200","vendor_id":"10de","product_id": "11b4"}
26. Nova΁ͷ௥ه(2) ίϯτϩʔϥʔϊʔυͷnova aliasͷઃఆ͕ඞཁɺޙ ड़͢Δflavor-key Ͱར༻͢Δɻ to /etc/nova/nova.confʹ pci_aliasesΛ௥ه pci_alias={“name”:”K4200”,"vendor_id":"10de","product_id":"11b4"}
27. Nova΁ͷ௥ه(3) ίϯτϩʔϥϊʔυʹͯPCIύεεϧʔϑΟϧλʔΛ novaʹ௥Ճɻ /etc/nova/nova.conf ʹΞϯμʔϥΠϯ෦෼௥Ճ scheduler_available_filters=nova.scheduler.filters.all_filters scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciP assthroughFilter scheduler_default_filters=DifferentHostFilter,RetryFilter,AvailabilityZoneFilter, RamFilter,CoreFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,Imag ePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,Aggre gateInstanceExtraSpecsFilter,PciPassthroughFilter
28. nova alias GPUΛ࢖͏ͨΊʹflavor-keyΛ௥ه͢ΔɻϑϨʔόʹset໊ Ͱطଘflavorʹର͠pciύεεϧʔͷͰܾΊͨGPUͷalias ໊ɺ࢖͍͍ͨGPUϢχοτ਺Λ௥Ճ͢Δɻ ্ه͸લड़·ͰͷաఔͰɺQuadro K4200Λ2Ϣχοτ௥Ճ͍ͯ͠ Δ͜ͱ͕લఏͳͷͰɺҎԼͷΑ͏ͳهड़ʹͳΔɻ pci_aliasΛK4200ͱ͍ͯ͠ΔͨΊҎԼͷهड़ͱͳΔɻ nova flavor-key $flavor_name set “pci_passthrough:alias”=“K4200:$amount_of_gpu”
29. ط஌ͷ໰୊
30. Ϋϥ΢υΠϝʔδͷ໰୊ Πϝʔδ͕GPUΛ࢖͏ʹ͸ͱͯ΋খ͘͞qemu-imgͰΠϝʔδͷϦα ΠζΛͯ͠΍Δඞཁ͕༗Δɻ CUDA driverΛΠϯετʔϧ͢Δʹ͸perl-packagesΛطଘͷΫϥ΢υ Πϝʔδʹ͸͍Εͯ΍Δඞཁ͕͋Δɻ CUDAυϥΠό͸.deb or .rpMύοέʔδͰ΋όΠφϦϑΝΠϧͰͳ͘ɺΠϯε τʔϧ࣌ʹιʔεΛmakeͰΠϯετʔϧ͢ΔܗΛͱ͍ͬͯΔ=.runϑΝΠϧͱม ΘΒͣɻ Nvidia͸কདྷͷϦϦʔεͰspecϑΝΠϧʹperlؔ࿈ϑΝΠϧͷ௥هΛ͢ΔܗͰfix ͢Δ༧ఆɻ 7.6Ҏ߱Ͱగਖ਼ͯ͘͠ΕΔɺͱ͍͍͕ͬͯͨ…(·ͩςετͯ͠ͳ͍ɻʣ
31. Windows as VDI Windows্ͷCUDA͸ͪΌΜͱΠϯετʔϧͰ͖Ε͹ૣ ͘ͳΔ͚Ͳ࣌ͨ·ΧΫΧΫͯ͠͠·͏ɻ vmͷdisk speedͷ໰୊ɺΤϑΣϝϥϧʹ͢ΔͳΓૣ͍σΟεΫΛ ࢖͏ͳΓͰͦͦ͜͜ղܾ͢Δ(SSD,NVMe or..etc) vm͸context switchͰr/wΛߦ͍ͬͯΔͷͰϔϏʔϫʔΫϩʔυ ͷCUDAͳΓ͸͜ͷΑ͏ͳΧΫΧΫঢ়ଶΛى͜͢Մೳੑ͕͋Δɻ ͜ͷ͋ͱ͸ޙ೚ʹ·͔͕ͤͨɺগ͠͸ΤϑΣϝϥϧͰվળͨ͠Β͍͠ ސ٬͸·ɺͳΜͱ͔ͳͬͨͱͱΓ͋͑ͣͷຬ଍͸΋Βͬͨɻ
32. ϥΠϒϚΠάϨʔγϣϯ ϥΠϒϚΠΫϨʔγϣϯ͸Ͱ͖ͳ͍ɺvm͕ݹ͍ϗετͷ઀ଓ৘ ใΛ੾Γ཭ͤͳ͍ɻݹ͍ϗετͷ৘ใΛѲͬͨ··ʹͳΔɻ ϫʔΫΞϥ΢ϯυ:nova.pci_devicesͷMySQLͷDB৘ใΛফ͠ ͯɺݹ͍ϗετΛ࠶ىಈ͢Δʼҙຯͳ͠ʂ | 2016-08-11 00:54:45 | 2016-08-19 04:58:01 | NULL | 0 | 45 | 21 | 0000:84:00.0 | 11b4 | 10de | type-PCI | pci_0000_84_00_0 | label_10de_11b4 | available | {} | NULL | NULL | 1 | <<-- old-host | 2016-08-11 00:54:45 | 2016-08-19 04:58:01 | NULL | 0 | 48 | 21 | 0000:88:00.0 | 11b4 | 10de | type-PCI | pci_0000_88_00_0 | label_10de_11b4 | available | {} | NULL | NULL | 1 | <<-- old-host
33. ·ͱΊ OpenStackͷಛघར༻͕Ͱ͖Δ͔Ͳ͏͔͸౔୆ͷOS࣍ୈ Ͱ͢ɻ ౔୆ͷOS͕μϝͳͷʹOpenStack͕..ͱ͍͏ͷ͸Πέͯͳ͍ਓ͕͍ ͏͓ݴ༿… αʔόܥOS͸ҙ֎ʹ͍Ζ͍ΖΠέͯ·ͤΜ…໰୊͋Γ͋ΓͰ͢ɻ GPUͷར༻͸ࠓ͸͜ͷύεεϧʔ͕ݱߦͰ͕͢ɺNvidia ΍Intel࣍ୈ͔ͱ(AMD͸…Ͳ͏͢ΔΜͩΖ͏…)
34. Thank you Masafumi Ohta @masafumiohta