Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tensorコアを使った PyTorch の高速化について

fam_taro
April 08, 2019

Tensorコアを使った PyTorch の高速化について

fam_taro

April 08, 2019
Tweet

More Decks by fam_taro

Other Decks in Technology

Transcript

  1. "HFOEB  5FOTPSίΞ JTԿ  1Z5PSDIͰ 5FOTPSίΞ࢖͏ʹ͸  ਪ࿦ ༧ଌ

    Ͱ࢖ͬͯΈͨ .%FU  ֶशͰ 5FOTPSίΞΛ࢖͍͍ͨ৔߹ 
  2. 5FOTPSίΞ JTԿ – CJUුಈখ਺఺ '1 ͷߦྻԋࢉʹಛԽͨ͠ܭࢉ༻ίΞ – 1Z5PSDI ౳ͷਂ૚ֶशϑϨʔϜϫʔΫΛ࢖͏ͱɺσϑΥϧτͰ͸ CJUුಈখ਺఺

    ʢ'1 Ͱॾʑͷܭࢉ͕ߦΘΕΔ – /7*%*"(16ͷ͏ͪ 7PMUBੈ୅Ҏ߱ͷҰ෦ʹ৐͍ͬͯΔ – 5VSJOHͰ΋ 359͡Όͳ͍ͱ৐ͬͯͳ͍ (59 5Jʹ͸৐͍ͬͯͳ͍ – 1BTDBM .BYXFMM ,FQMFSʹ͸ແ͍ ແ࣊൵ – (16ྫ – 5*5"/7 – (F'PSDF359γϦʔζ    5J – 5*5"/359 – 5FTMB7  ($1ͰࢼͤΔ – ࠓޙΑΓීٴ͢Δͱݸਓతʹࢥͬͯ·͢ – ࢀߟϦϯΫ – IUUQTXJLJXJLJKQOWJEJBWHB(16&##&"&#&""&"" – IUUQTKBXJLJQFEJBPSHXJLJ/7*%*"@(F'PSDF  https://www.nvidia.com/content/apac/gtc/ja/pdf/2017/1040.pdf
  3. 5FOTPSίΞ JTԿ – 5FOTPSίΞ࢖Θͳ͍ͱ '1ʹͯ͠΋଎౓͸ ഒఔ౓ – 5FOTPSίΞ࢖͏ͱ '1ʹͨ͠Β ഒ

    Ҏ্ʂ – ͨͩ͠࢖͏ͨΊʹ͸੍໿͕ଟ͍ʜ – $6%"Ҏ߱ – DV%// Ҏ߱ – '1Ͱѻ͏͜ͱΛ໌ه ϥΠϒϥϦͷίϚϯυ౳Ͱ – $POW૚ͷೖྗɾग़ྗνϟϯωϧ਺͕ ͷഒ਺ – શͯͷ૚͕ରԠ͍ͯ͠ΔΘ͚Ͱ͸ͳ͍ Ή͠ΖҰ෦͔͠ – ͋͘·Ͱݱ࣌఺ͳͷͰࠓޙ͸؇࿨͞ΕΔ͔΋ʜ  https://www.slideshare.net/NVIDIAJapan/chainer-tensor-fp16
  4. 1Z5PSDIͰ 5FOTPSίΞ࢖͏ʹ͸ – '1Λ࢖͏͜ͱΛ໌ه͢Ε͹ϑϨʔϜϫʔΫ͕উखʹ ࢖ͬͯ͘ΕΔ ͜ͱ͕ଟ͍ – ࠷ۙͷόʔδϣϯʹ͠ͳ͍ͱ͍͚ͳ͍͕ʜ – 1Z5PSDIͰ͸ʜ

    – .PEFMͱ *OQVUʹର͠ lIBMG zΛ෇͚Δ – ൒ਫ਼౓ʹ͢Δͱ͍͏ҙຯ '1ʹ͢Δ – 0VUQVU͸ '1ͱ͸ݶΒͳ͍ͷͰ஫ҙ – $POW૚ͷೖྗͱग़ྗνϟϯωϧ਺Λ ͷഒ਺ʹ͢Δ – ஍ຯʹ੍͖͍ͭ໿ – 1SFUSBJONPEFMͩͱ͏·͘࢖͑ͳ͍έʔε͕ग़ͯ͘Δ – ໌ࣔతʹ 5FOTPSίΞΛ࢖͏Α͏ʹ͸Ͱ͖ͳ͍ Ά͍ʁ 
  5. ਪ࿦ ༧ଌ Ͱ࢖ͬͯΈͨ .%FU – .%FUͱ͸ – ࿦จIUUQTRJKJF[IBPHJUIVCJPJNHTNEFUQEG – ࣮૷IUUQTHJUIVCDPNRJKJF[IBP.%FU

    – ࠷ۙग़͖ͯͨݕग़Ϟσϧ – :0-0WΑΓ଎ͯ͘ਫ਼౓͕ߴ͍ Β͍͠ – ࣗ෼ͷσʔλͰ͸ֶशશ͘Ͱ͖ͯͳ͍Ͱ͕͢! – ࣮ݧ৚݅ – $0$0EBUBTFU %FUFDUJPOͰ͸ϝδϟʔͳ΍ͭ – ֶशࡁΈϞσϧ Y Λ࢖༻ – ެࣜϦϙδτϦʹ͋Δ UFTUQZ Λ࢖༻ – NPEFM JOQVU ͷ࣌ؒͱޙॲཧ /.4 ͷ࣌ؒΛܭଌ – ଍ͨ࣌ؒ͠Λ࢖ͬͯ '14Λग़͢ ߴ͍΄Ͳྑ͍ –   ຕͷσʔλʹରͯ͠ਪ࿦ ༧ଌ Λͯ͠ฏۉ࣌ؒΛͱΔ – ݩ࿦จ͸  ຕͰऔ͍ͬͯͨ – Ϟσϧͷਫ਼౓͸ N"1 
  6. ਪ࿦ ༧ଌ Ͱ࢖ͬͯΈͨ .%FU – ࣮ߦ࣌ؒ΍ 5FOTPSίΞΛ࢖͍ͬͯΔ͔֬ೝ͢ΔͨΊʹ࢖ͬͨίϚϯυ – UJNFUJNF ؔ਺

    – (16བྷΉͨΊԼهͷํ͕ྑͦ͞͏Ͱ͕͢ࠓճ͸ UJNF ؔ਺Ͱ࣌ؒͱΓ·ͨ͠! – UPSDIDVEB&WFOU FOBCMF@UJNJOH5SVF – QSPGJMFS – QZUIPONUPSDIVUJMTCPUUMFOFDL UFTUQZ ddd – 1Z5PSDI ͷఏڙ͢Δ QSPGJMFS – ݕग़ͷޙॲཧ ࠓճ͸ /.4 ͕ϘτϧωοΫʹͳ͍ͬͯͳ͍͔֬ೝ͢ΔͨΊʹ࢖༻ – OWQSPG QZUIPOddd – /7*%*"ͷఏڙ͢Δ QSPGJMFSίϚϯυɻ5FOTPSίΞ࢖͍ͬͯΔ͔ݟΕΔ – OWDD ddd ະ࢖༻ – OWQSPG ͷίϚϯυΛ (6*ͰϦονʹݟΕΔΒ͍͠ɻͨͩ͠ΞϓϦαΠζσΧ͍ ਺(# 
  7. ਪ࿦ ༧ଌ Ͱ࢖ͬͯΈͨ .%FU – ࣮ߦ݁Ռ  (16 N"1 ਫ਼౓

    %FUFDUUJNF QFS JNBHF<NT> /NT UJNF 1FS JNBHF<NT> 5PUBM<NT> '14 ݩ࿦จ ϦϙδτϦ্ͷ஋ 5JUBO9 1"4$"-    1Z5PSDI ެࣜࢦఆόʔδϣϯ 5JUBO7      1Z5PSDI '1 5JUBO7      1Z5PSDIQPTU 5JUBO7      1Z5PSDIQPTU '1 5JUBO7     
  8. ਪ࿦ ༧ଌ Ͱ࢖ͬͯΈͨ .%FU – ࣮ߦ݁Ռ  (16 N"1 ਫ਼౓

    %FUFDUUJNF QFS JNBHF<NT> /NT UJNF 1FS JNBHF<NT> 5PUBM<NT> '14 ݩ࿦จ ϦϙδτϦ্ͷ஋ 5JUBO9 1"4$"-    1Z5PSDI ެࣜࢦఆόʔδϣϯ 5JUBO7      1Z5PSDI '1 5JUBO7      1Z5PSDIQPTU 5JUBO7      1Z5PSDIQPTU '1 5JUBO7      l1Z5PSDIʹͯ͠z ͔ͭ l'1࢖͏zͱ࠷΋଎͘ͳΔ ˠ ࠷ॳͷ ഒ ͔Ζ͏ͯ͡ݩ࿦จͷ '14Λ௒͑ͨ ༧ଌ࣌ͷΈͳΒ '1ʹͯ͠΋ ਫ਼౓͸มΘΒͳ͍ 1Z5PSDIͷόʔδϣϯ্͛ͨΒ଎͘ ͳΔ͠ɺ'1ʹͯ͠΋଎͘ͳΔ
  9. ֶशͰ 5FOTPSίΞΛ࢖͍͍ͨ৔߹ – ͱΓ͋͑ͣશ෦ '1ʹ͢Ε͹͍͍ΜͰ͠ΐʁˠ /Pʂ – ޯ഑ܭࢉ࣌ʹ͔ͳΓӨڹ͢Δ ޯ഑͕ফ͑Δ৔߹΋͋Δ –

    ˠ ਫ਼౓͕େ͖͘Լ͕Δ৔߹͕͋Δ – .JYFE1SFDJTJPO5SBJOJOH – ࢀߟϦϯΫIUUQTXXXTMJEFTIBSFOFU/7*%*"+BQBODIBJOFSUFOTPSGQ – '1ͱ '1ʹΑΔܭࢉΛ NJYֶͨ͠शํ๏ – ۩ମతʹ͸ҎԼͷΑ͏ͳςΫχοΫ͕ඞཁ – ϩεεέʔϦϯά ϩε஋ΛཁॴཁॴͰεέʔϧ͋ͬ΀ͯ͠ޯ഑ফࣦΛ؇࿨ – '1΢ΣΠτߋ৽ 'PSXBSEͱ #BDLXBSE͸ '1 6QEBUFͰ '1Λ࢖༻ – ্هΛશͯࣗ෼Ͱ΍Δͱେมʂ ֶͼ͸͋Δͱࢥ͍·͕͢ʜ – ˠ BQFYΛ࢖͏ͱൺֱతखܰʹͰ͖Δ IUUQTHJUIVCDPN/7*%*"BQFY – /7*%*"͕ఏڙ͢Δ 1Z5PSDI༻ "VUPNBUJD.JYFE1SFDJTJPO ".1 πʔϧ – ݩͷίʔυʹର͠਺ߦ଍͚ͩ͢Ͱ .JYFE1SFDJTJPO5SBJOJOHͰ͖Δͱͷ͜ͱ – ͨͩ͠ JOTUBMM࣌͸ $6%"΍ 1Z5PSDIͷόʔδϣϯʹؾΛ͚ͭͳ͍ͱ͍͚ͳ͍ 
  10. 3FGFSFODFT – <೥൛ ػցֶशϋʔυ΢ΣΞͷ4UBUFPGUIF"SUΛߟ͑Δ d$16 (16 516Λఴ͑ͯd 2JJUB> IUUQTRJJUBDPNBSVUFNBJUFNTGCB –

    <5SBJOJOH/FVSBM/FUXPSLTXJUI.JYFE1SFDJTJPO /7*%*"> IUUQPOEFNBOEHQVUFDIDPOGDPNHUDUBJXBOQEG @*OUFSOBM4QFBLFS@.JDIBFM$BSJMMJ@1%''PS4IBSJOHQEG – <$IBJOFS Ͱ 5FOTPSίΞ GQ Λ࢖͍͜ͳ͢> IUUQTXXXTMJEFTIBSFOFU/7*%*"+BQBODIBJOFSUFOTPSGQ – <$IBJOFSʹ͓͚Δਂ૚ֶशͷߴ଎Խ> IUUQTXXXOWJEJBDPNDPOUFOUBQBDHUDKBQEGQEG – <70-5""/%563*/("3$)*5&$563&"/%1&3'03."/$&015*.*;"5*0/ /7*%*"> IUUQTXXXOWJEJBDPNDPOUFOUBQBDHUDKBQEGQEG – <5SBJOJOHXJUI.JYFE1SFDJTJPO%FFQ-FBSOJOH4%,%PDVNFOUBUJPO> IUUQTEPDTOWJEJBDPNEFFQMFBSOJOHTELNJYFE QSFDJTJPOUSBJOJOHJOEFYIUNMQZUPSDI – <(16࢓༷Ұཡද  /7*%*"(F'PSDF 8JLJ > IUUQTXJLJXJLJKQOWJEJBWHB(16&##&"&#&""&""