分析前処理で手軽に並列処理するために守っておくと良いこと

by yasubei

Slide 1

Slide 1 text

ࠓ ि ͷ Θ ͔ Β Μ ઌ ੜ Ϛϧνϓϩηε ॲཧ͕ Θ͔ΒΜ 1Z%BUB0LJOBXBΦʔΨφΠβʔ ΍͢΂͐

Slide 2

Slide 2 text

ࣗݾ঺հ :BTVCFJ ΍͢΂͐ ZBTVCFJUXJ ZBTVIBSVTV[VLJ IUUQTIVHLVODPN ΤϯδχΞͰ͢ɻ ࡳຈͰήʔϜ։ൃ೥ɹˠɹԭೄͰ*P5 8։ൃ ෼ੳͳͲΛ೥ )VHLVOͱ)VHLVO%BUBͱ͍͏ձࣾ΍ͬͯ·͢ɻ اۀ͞Μͱֶੜ͞ΜͱҰॹʹ࢓ࣄͯ͠8JO8JO8JOͳؔ܎Λங͘ͷ͕޷͖Ͱ͢ɻ ʮ౳෼ͷՖՇʯΛม૷່ͨ͠ͷਖ਼ղΛ౰ͯͳ͕ΒಡΉͷ͕࠷ۙͷϚΠϒʔϜ ΧϥΦέͰ།Ұ๙ΊΒΕͨۂ͸ʮਓੜ͍Ζ͍Ζʯ

Slide 3

Slide 3 text

ͱ͍͏Θ͚ͰͭࢠΫʙΠζ

Slide 4

Slide 4 text

Θ͔Δਓ ɾТɾ ϊγ

Slide 5

Slide 5 text

ਖ਼ղ

Slide 6

Slide 6 text

΋͏ͪΐͬͱ؆୯ͳ΍ͭ

Slide 7

Slide 7 text

͜Μͳײ͡Ͱ͢

Slide 8

Slide 8 text

ࢥΘͣ෼ྨͨ͘͠ͳͬͨํ͸ίνϥ IUUQTTQFBLFSEFDLDPNOZBOLPBOENBNFOPLJXVUJKJBODIVGBMTFMJTIJUPNEFUOJZPSV[VJYJOGBMTFBQVSPUJ 63-

Slide 9

Slide 9 text

ࠓ೔ͷ͓୊

Slide 10

Slide 10 text

ࠓ೔ͷ͓୊ Ϛϧνϓϩηεॲཧ

Slide 11

Slide 11 text

ʮͦʔΜͳϞϯ஌ͬͯΔϤʙʯ ͱ͍͏ํ ֶੜͰ΋ଟ͍Μ͡Όͳ͍͔ͱࢥ͍·͢

Slide 12

Slide 12 text

ͦΕ͕ͳʔ

Slide 13

Slide 13 text

ҙ֎ͱ૊Ίͳ͍ΜͰ͢Α ɾТɾʆ

Slide 14

Slide 14 text

஌ࣝ͸͋Δ͚Ͳ ؆୯ʹ૊Ή΍ΓํΛ஌Βͳ͍ਓ͕ ҙ֎ͱଟ͍ͳ͊ʙͬͯࢥ͏͜ͱ͕ଟ͍ͷͰ ͜ΕΛकΔͱ؆୯ʹ૊ΊΔΑͬͯ ࿩Λ͍ͨ͠ͱࢥ͍·͢ɻ ʢ˞ͨͩ͠ࠓ೔৮ΕΔͷ͸෼ੳ༻Ͱ͢ʣ

Slide 15

Slide 15 text

:BTVCFJ ΍͢΂͐ ඇಉظϓϩάϥϛϯάͷܦݧ ZBTVCFJUXJ ZBTVIBSVTV[VLJ IUUQTIVHLVODPN ɾֶੜͷ͜ΖόΠτͰϔουϚ΢ϯτσΟεϓϨΠ͔Βͷ৴߸ॲཧΛड৴͢Δ෦෼Λ ɹϚϧνεϨοσΟϯάʢ8JOʣͰهड़ͨ͠ͷ͕࠷ॳ ɹྈੜ׆Ͱྈʹωοτ͕ͳ͘ɺ৘ใऩूʹۤ࿑ͨ͠ ɾ೥͝Ζʹॻ͍ͨωοτରઓ༻ͷϓϩάϥϜ͸$Ͱ-JOVY্Ͱ૊ΜͰ·ͨ͠ ɹ΄΅ΦϯϝϞϦͰ௒ߴ଎Ͱͨ͠ ɹฏۉԠ౴଎౓͸NTҎԼ ɾ࠷ۙͰ͸ϏοάσʔλͷલॲཧΛߴ଎Խ͢ΔͨΊʹฒྻԽΛ༻͍Δࣄ͕ଟ͍ ɾԿؾʹKTͰ΋࢖༻ͯ͠·͢Ͷ ɹ7VFKT࢖͏Ҋ͕݅͋Γɺશ෦ͷ"1*ΞΫηε͕ඇಉظ

Slide 16

Slide 16 text

"HFOEB ର৅ऀ कΔͱྑ͍ϧʔϧ ूܭॲཧΛBQQMZ Ͱ૊ΜͰ͓͘ Ξ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ ߦ୯ҐͷฒྻԽ͸EBTLΛ࢖͏ େ͖ͳ୯ҐͷฒྻԽ͸KPCMJCΛ࢖͏ σϞʢ͕࣌ؒ͋Ε͹ʣ ·ͱΊ

Slide 17

Slide 17 text

ର৅ऀ Θ ͔ Β Μ

Slide 18

Slide 18 text

ର৅ऀ ɾͪΐͬͱϏοάσʔλѻͬͯΔ͚ͲQBOEBTॏ͍ ɾฒྻॲཧॻ͍ͯߴ଎Խ͍͚ͨ͠ͲΠϝʔδ͔ͭΈʹ ࣮ͯ͘͘૷ʹֻ͕͔࣌ؒͬͯ͠·͏ ɾ਺ࣜಘҙ͚ͩͲ1ZUIPOۤखͳ਺ֶՊͷํ

Slide 19

Slide 19 text

ର৅ऀͷ1ZUIPOϨϕϧ ೖ໳ɹɹ#FHJOOFS ॳڃɹɹ&MFNFOUBSZ தڃɹɹ*OUFSNFEJBUF ४্ڃɹ6QQFS*OUFSNFEJBUF ্ڃɹɹ"EWBODFE ͨͿΜ͜ͷล 1ZUIPO͸ීஈ࢖ͬͯͯ#JH%BUB΋ѻͬͯΔ͚Ͳ ฒྻԽͷ࣮૷্͕ख͘ߦͬͯͳ͍ਓ޲͚

Slide 20

Slide 20 text

ݸਓతʹΘ͔Γ΍͍͢#JH%BUBͷఆٛ &YDFMͰ։͚ͳ͘ͳͬͨΒ #JH%BUB ˞΋ͪΖΜ*OUFSOBM3FGFSFODFͰ͢

Slide 21

Slide 21 text

कΔͱྑ͍ϧʔϧ Θ ͔ Β Μ

Slide 22

Slide 22 text

कΔͱྑ͍ϧʔϧ ɾQBOEBTΛ࢖͏ ɾूܭॲཧΛBQQMZ Ͱ૊ΜͰ͓͘ ɾΞ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ ɾߦ͝ͱͷॲཧͷฒྻԽ͸EBTLΛ࢖͏ ɾେ͖ͳ୯ҐͷฒྻԽ͸KPCMJCΛ࢖͏

Slide 23

Slide 23 text

QBOEBTΛ࢖͏ ·͊ͨͿΜΈΜͳ࢖ͬͯΔͷͰίϨ͸͍͍Ͱ͢ΑͶ

Slide 24

Slide 24 text

ूܭॲཧ͸BQQMZ Ͱ૊ΜͰ͓͘ Θ ͔ Β Μ

Slide 25

Slide 25 text

ूܭॲཧ͸BQQMZ Ͱ૊ΜͰ͓͘ ɾEBTLͰͷฒྻԽ͕༰қʹͳΓ·͢ ɾ໊ݴʮGPSจ࢖ͬͨΒෛ͚ͩͱࢥ͍ͬͯΔʯ ɹɹɹʮࢲͷQBOEBT஗͗͢ʯ ɹͦ΋ͦ΋JUUFSPXTͱ͔࢖Θͳ͍Ͱ͢ΑͶʁʢἤΓ

Slide 26

Slide 26 text

ूܭॲཧ͸BQQMZ Ͱ૊ΜͰ͓͘ GPSจʢΞϯνύλʔϯʣ ॳΊ͔Βʮॏ͘ͳΓͦ͏ͩͳ͊ʯͱࢥ͍ͬͯΔՕॴʹ͸ GPSจ͸࢖Θͳ͍͜ͱʢܰͦ͏ͳͱ͜Ζ͸GPSͰ΋ྑ͍ʣ IPHF@MJTU<> GPSJOEFY SPXJOEGJUUFSPXT IPHF@EBUBԿΒ͔ͷॏ͍ॲཧ IPHF@MJTUBQQFOE IPHF@EBUB BQQMZจʢྑ͍ύλʔϯʣ BQQMZʹ͓ͯ͘͜͠ͱࣗମ΋ߴ଎Խʹܨ͕Δ͚Ͳ ͦ΋ͦ΋ฒྻԽ͢Δʹ͸ؔ਺Խ͓ͯ͘͠ͱָ EFGBQQMZ@GVOD SPX IPHF@EBUBԿΒ͔ͷॏ͍ॲཧ SFUVSOIPHF@EBUB IPHF@MJTUEGBQQMZ BQQMZ@GVOD BYJT #BE #FUUFS GPSจͷྫ BQQMZจͷྫ ˞͋͘·ͰCFUUFSɻCFTUΛ໨ࢦ͢ʹ͸͍Ζ͍Ζͳ΍Γํ͕͋ Δ͕ɺݸਓతʹ͸࢖͍উखͷྑ͍BQQMZ͕ྑ͍ɻ

Slide 27

Slide 27 text

Ξ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ Θ ͔ Β Μ

Slide 28

Slide 28 text

Ξ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ ͬŋТŋ ͬŲƄŖƃ ͨͱ͑͹Ұ͔݄෼ͷσʔλΛग़ྗ͢Δ৔߹ ԿΒ͔ͷॲཧ ෼͚Δ

Slide 29

Slide 29 text

Ξ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ ͬŋТŋ ͬŲƄŖƃ ϓϩηε͝ͱʹग़ྗઌΛ෼͚Δͷ͕ཧ૝ ԿΒ͔ͷॲཧ ͬŋТŋ ͬŲƄŖƃ ͬŋТŋ ͬŲƄŖƃ

Slide 30

Slide 30 text

Ξ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ ̑ͭࢠͰΠϝʔδ͢Δͱ͜Μͳײ͡ ݱࡏ ະདྷ աڈ

Slide 31

Slide 31 text

Ξ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ ཧ༝ ɾ*0ͷϩοΫɺഉଞ੍ޚ͕ཁΒͳ͘ͳΔ ɹˠͭ·Γɺ஗Ԇ΍σουϩοΫΛߟྀ͢Δඞཁ͕ͳ͘ͳΔ ɹɹɹϑΝΠϧ*0͸ϝϞϦΑΓഒ஗͍ ɹɹɹωοτϫʔΫ*0͸΋ͬͱ஗͍

Slide 32

Slide 32 text

Ξ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ ͬŋТŋ ͬŲƄŖƃ Ξϯνύλʔϯ ԿΒ͔ͷॲཧ ෼͚ͳ͔ͬͨ ৔߹

Slide 33

Slide 33 text

Ξ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ ͬŋТŋ ͬŲƄŖƃ ϓϩηεΛ෼͚ͯ΋஗Ԇ͕ൃੜͯ͠͠·͏ ԿΒ͔ͷॲཧ ͬŋТŋ ͬŲƄŖƃ ͬŋТŋ ͬŲƄŖƃ

Slide 34

Slide 34 text

ߦ͝ͱͷߴ଎Խ͸EBTLΛ࢖͏ Θ ͔ Β Μ

Slide 35

Slide 35 text

ߦ͝ͱͷߴ଎Խ͸EBTLΛ࢖͏ EBTLͱ͸ʁ ɾ%BTLͱ͸Ϛϧνϓϩηεͷॲཧ͕؆୯ʹͰ͖Δ 1ZUIPOͷϥΠϒϥϦͰ͢ɻ

Slide 36

Slide 36 text

ߦ͝ͱͷߴ଎Խ͸EBTLΛ࢖͏ ɾBQQMZΛ࢖͍ͬͯΔͷͰ͋Ε͹؆୯ͳमਖ਼ͰEBTLԽͰ ͖·͢ɻ ɾEBTLʹ͸ϚϧνϓϩηεͰฒྻԽ͢Δػೳ͕͋Γɺ؆ ୯ʹฒྻԽͰ͖·͢ɻ ɾ࢖༻͢Δ$16਺Λࢦఆ͢Δ͜ͱ΋ग़དྷ·͢ɻҾ਺Ͱݸ ਺Λࢦఆ͢Δ͚ͩͰඇৗʹखܰͰ͢ɻ

Slide 37

Slide 37 text

ߦ͝ͱͷߴ଎Խ͸EBTLΛ࢖͏ EBTLԽ ίΞ࢖༻͢ΔEBTL༻%BUB'SBNFΛ࡞੒ JNQPSUEBTLEBUBGSBNFBTEE EEGEEGSPN@QBOEBT EG OQBSUJUJPOT EFGBQQMZ@GVOD SPX IPHF@EBUBԿΒ͔ͷॏ͍ॲཧ SFUVSOIPHF@EBUB IPHF@MJTUEGBQQMZ BQQMZ@GVOD BYJT NFUBܕΛࢦఆ DPNQVUF TDIFEVMFSQSPDFTTFT EBTLͷྫ QBOEBTͷBQQMZΛ࢖༻͢Δྫ લड़ͷBQQMZͱಉ͡ϓϩάϥϜͰ͢ɻ EFGBQQMZ@GVOD SPX IPHF@EBUBԿΒ͔ͷॏ͍ॲཧ IPHF@MJTUEGBQQMZ BQQMZ@GVOD BYJT BQQMZจͷྫ GSPN@QBOEBTͰม׵ͯ͠ BQQMZ ͢Δ͚ͩ

Slide 38

Slide 38 text

ߦ͝ͱͷߴ଎Խ͸EBTLΛ࢖͏ ҰԠࣄྫͱͯ͠͸ ɹສ݅͘Β͍ͷσʔλΛߜΓࠐΈͳ͕Βूܭ͢Δॲཧ ɹɾTFD ɹɹɹˣEBTLͰฒྻԽˍίΞͷ$16Ͱϑϧճస ɹɾTFD ɹഒΑΓ଎͘ͳͬͨͷ͸Ṗ

Slide 39

Slide 39 text

ߦ͝ͱͷߴ଎Խ͸EBTLΛ࢖͏ Ξϯνύλʔϯ ɹจࣈྻ಺ͷ୯ޠΛΧ΢ϯτ͢Δॲཧ ɹɾTFD ɹɹɹˣEBTLͰฒྻԽˍίΞͷ$16Ͱϑϧճస ɹɾTFD ɹϚϧνϓϩηεԽ͢Δίετ΋݁ߏߴ͍ͷͰ ɹ୯७ͳॲཧ͸஗͘ͳΔ͜ͱ͕ଟ͍Ͱ͢ɻ

Slide 40

Slide 40 text

େ͖ͳ୯ҐͷฒྻԽ͸+PCMJCΛ࢖͏ Θ ͔ Β Μ

Slide 41

Slide 41 text

େ͖ͳ୯Ґͷߴ଎Խ͸+PCMJCΛ࢖͏ +PCMJCͱ͸ʁ ɾ+PCMJCͱ͸Ϛϧνϓϩηεͷॲཧ͕؆୯ʹͰ͖Δ 1ZUIPOͷϥΠϒϥϦͰ͢ɻ

Slide 42

Slide 42 text

େ͖ͳ୯Ґͷߴ଎Խ͸+PCMJCΛ࢖͏ ଞͷฒྻॲཧΛ࢖Θͳ͍ཧ༝ ɾ4QBSLˠΊΜͲ͍ʢ޻਺Λަব͢Δඞཁ͕͋Δʣ ɾBTZODΩʔϫʔυ͸$Ͱେݏ͍ͩͬͨ ʢ1ZUIPOͰ͸৯Θͣݏ͍ʣ ɾNVMUJQSPDFTTJOH1SPDFTT͸एׯखܰ͞ʹ͚ܽΔ ʢ࢖͑ͳ͍͜ͱ͸ͳ͍ʣ ͋͘·Ͱݸਓతͳҙݟ

Slide 43

Slide 43 text

େ͖ͳ୯Ґͷߴ଎Խ͸+PCMJCΛ࢖͏ EBTLԽɹϓϩηεͰॲཧ͠·͢ ίΞ࢖༻͢ΔEBTL༻%BUB'SBNFΛ࡞੒ JNQPSUEBTLEBUBGSBNFBTEE EEGEEGSPN@QBOEBT EG OQBSUJUJPOT EFGBQQMZ@GVOD SPX IPHF@EBUBԿΒ͔ͷॏ͍ॲཧ SFUVSOIPHF@EBUB IPHF@MJTUEGBQQMZ BQQMZ@GVOD BYJT NFUBܕΛࢦఆ DPNQVUF TDIFEVMFSQSPDFTTFT EBTLͷྫ KPCMJCԽYͰϓϩηεͰॲཧ͠·͢ EGΛͭʹ෼ׂ GSPNKPCMJCJNQPSU1BSBMMFM EFMBZFE 4&1"3"5&@$06/5 TFQ@TUFQJOU MFO EG 4&1"3"5&@$06/5 EG@MJTU<> GPSTUFQJOSBOHF MFO EG TFQ@TUFQ EG@MJTUBQQFOE EG KPCMJC༻ͷؔ਺ఆٛ EFGKPCMJC@GVOD EG EEGEEGSPN@QBOEBT EG OQBSUJUJPOT EFGBQQMZ@GVOD SPX IPHF@EBUBԿΒ͔ͷॏ͍ॲཧ SFUVSOIPHF@EBUB IPHF@MJTUEGBQQMZ BQQMZ@GVOD BYJT NFUBܕΛࢦఆ DPNQVUF TDIFEVMFSbQSPDFTTFT SFUVSOIPHF@MJTU KPCMJC༻ͷؔ਺ఆٛ GVHB@MJTU1BSBMMFM O@KPCT4&1"3"5&@$06/5 GPSTUFQJO SBOHF 4&1"3"5&@$06/5 > +PCMJCͷྫ एׯख͚ؒͩͲEBTLͷॲཧ͸ ΄΅ྲྀ༻Ͱ͖Δ

Slide 44

Slide 44 text

େ͖ͳ୯Ґͷߴ଎Խ͸+PCMJCΛ࢖͏ ͬŋТŋ ͬŲƄŖƃ +PCMJC ͬŋТŋ ͬŲƄŖƃ ͬŋТŋ ͬŲƄŖƃ ͬŋТŋ ͬŋТŋ ͬŋТŋ ͬŋТŋ ͬŋТŋ %BTL

Slide 45

Slide 45 text

ݱࡏ ະདྷ աڈ େ͖ͳ୯Ґͷߴ଎Խ͸+PCMJCΛ࢖͏ ͭࢠͷ৔߹

Slide 46

Slide 46 text

େ͖ͳ୯Ґͷߴ଎Խ͸+PCMJCΛ࢖͏ EBTLͱ+PCMJCͲͬͪΛ࢖͏΂͖͔ ɾ%BUB'SBNF಺ͷߦ୯Ґͷॲཧ͸EBTL ɾ*0ͷେ͖ͳॲཧ΍Կ෼΋ֻ͔Δॲཧ͸+PCMJC ɾEBTLͷํ͕࣮૷ίετ͸͍҆ ɾएׯ࣮૷ʹ࣌ؒΛֻ͚ͯ΋͞ΒʹฒྻԽ͍ͨ͠ͱ͖͸+PCMJC ͍͍ͩͨ͜Μͳײ͡ ͋͘·Ͱݸਓతͳҙݟ

Slide 47

Slide 47 text

σϞʢ͕࣌ؒ͋Ε͹ʣ Θ ͔ Β Μ

Slide 48

Slide 48 text

·ͱΊ Θ ͔ Β Μ

Slide 49

Slide 49 text

·ͱΊ ूܭॲཧ͸BQQMZ Ͱ૊ΜͰ͓͘ Ξ΢τϓοτઌ͸෼ׂ͓ͯ͘͠ ߦ͝ͱͷॲཧͷฒྻԽ͸EBTLΛ࢖͏ େ͖ͳ୯ҐͷฒྻԽ͸KPCMJCΛ࢖͏ ʙ Λ͓ͯ͘͠ͱ ͸௨ৗͷ޻਺ͷൣғ಺ ͸ʮͪΐͬͱॲཧ͕ॏͯ͘ฒྻԽதͰ͢ʔʯͱ্࢘ʹݴͬͯ औΓ૊Ί͹ྑ͍಺༰ʢ͍͍ͤͥʙ࣌ؒఔ౓Ͱ࣮૷Ͱ͖Δ ͷͰ͓͢͢Ί