Slide 1

Slide 1 text

ΧϥϜφϑΥʔϚοτͷ͖΄Μ !DIJF

Slide 2

Slide 2 text

໨࣍ •  ࣗݾ঺հ •  Ϗοάσʔλج൫ʹ͓͚ΔΧϥϜφϑΥʔϚοτ •  "QBDIF1BSRVFUͷ͖΄Μ –  ωετ͞Εͨσʔλʹదͨ͠σʔλϨΠΞ΢τ –  Ϧιʔεར༻ͷ࠷దԽ –  ૬ޓӡ༻ੑ •  03$ͱ1BSRVFUͷҧ͍ •  ·ͱΊ

Slide 3

Slide 3 text

ࣗݾ঺հ $IJF)BZBTIJEB !DIJF 3FUUZ*OD 4PGUXBSF&OHJOFFS%BUB4DJFOUJTU "84ྺ͸೥΄Ͳɻ 4QBSL΍;FQQFMJOͳͲʹίϯτϦϏϡʔτɻ ػցֶश)BEPPQ4QBSL4DBMB1ZUIPO%#WJN ম೑ϐΞϊςχεεϊϘ 機械学習& 分析基盤構築やってます!

Slide 4

Slide 4 text

೔ຊ࠷େڃɺ໊࣮ܕάϧϝαʔϏε

Slide 5

Slide 5 text

3FUUZͷ෼ੳج൫ͷ঺հ

Slide 6

Slide 6 text

ൃද಺༰ʹ͍ͭͯ •  Ϗοάσʔλج൫Ͱ͸ɺσʔλͷϑΥʔϚοτʹΧϥϜφ ϑΥʔϚοτΛ࢖͏ͱ͍͏ͷ͕Ұͭͷఆੴͱͳ͍ͬͯ·͢ɻ •  ຊൃදͰ͸ɺΧϥϜφϑΥʔϚοτͷಛ௃Λɺ1BSRVFUΛ ϕʔεʹղઆ͠·͢ɻ •  ͳͥΧϥϜφϑΥʔϚοτΛ࢖͏΂͖ͳͷ͔ɺͲͷΑ͏ʹ ࢖͏΂͖ͳͷ͔Λཧղͯ͠΋Β͑ͨΒΑ͍ͳɺͱࢥ͍·͢ɻ Amazon EMR Amazon Athena Amazon Redshi1

Slide 7

Slide 7 text

Ϗοάσʔλج൫ʹ͓͚Δ ΧϥϜφϑΥʔϚοτ

Slide 8

Slide 8 text

•  ςΩετϑΥʔϚοτ •  ߦࢦ޲ϑΥʔϚοτ •  ྻࢦ޲ʢΧϥϜφʣϑΥʔϚοτ Ϗοάσʔλج൫Ͱ࢖ΘΕΔϑΥʔϚοτ CSV { JSON } σʔλϕʔεʹΑ͘ ࢖ΘΕΔϑΥʔϚοτ

Slide 9

Slide 9 text

•  ߦࢦ޲ϑΥʔϚοτ ߦࢦ޲ϑΥʔϚοτͱྻࢦ޲ϑΥʔϚοτ •  ྻࢦ޲ϑΥʔϚοτ colA colB colC 1 aaa 2017-04-01 2 bbb 2017-04-02 3 ccc 2017-04-03 4 ddd 2017-04-04 5 eee 2017-04-05 colA colB colC 1 aaa 2017-04-01 2 bbb 2017-04-02 3 ccc 2017-04-03 4 aaa 2017-04-04 5 ccc 2017-04-05 行方向に連続してデータを保持 ・ 行単位でデータを操作することの多い OLTP処理に有利 列方向に連続してデータを保持 ・ 特定の列のみ取得する分析用途に有利 ・ 列のカーディナリティが低い場合、 圧縮率が高くなる σʔλ֨ೲ୯Ґ 一般的に分析用途のビッグデータ基盤はこっちが向いている

Slide 10

Slide 10 text

•  03$ —  "QBDIF)JWFͷͨΊʹ࡞ΒΕͨϑΥʔϚοτ —  "QBDIFͷτοϓϓϩδΣΫτ —  )JWF1SFTUP4QBSLͳͲ͕αϙʔτ •  1BSRVFU —  5XJUUFS͕ࣾ։ൃɺ$MPVEFSB΋։ൃʹେ͖ؔ͘ΘΔ —  "QBDIFͷτοϓϓϩδΣΫτ —  4QBSL1SFTUP%SJMM*NQBSB)JWFͳͲ͕αϙʔτ ओཁͳ044ͷྻࢦ޲ϑΥʔϚοτ ࠓճ͸1BSRVFUΛϕʔεʹઆ໌͠·͢

Slide 11

Slide 11 text

"QBDIF1BSRVFUͷ͖΄Μ

Slide 12

Slide 12 text

1BSRVFU •  ೥ʹ(PPHMF͕ൃදͨ͠%SFNFMͷ࿦จͷ SFDPSETISFEEJOHBOEBTTFNCMZBMHPSJUINͷ಺༰Λ ϕʔεʹ։ൃ͞Εͨ Ҿ༻ɿIUUQTTUBUJDHPPHMFVTFSDPOUFOUDPNNFEJBSFTFBSDIHPPHMFDPNKBQVCTBSDIJWFQEG

Slide 13

Slide 13 text

•  ωετ͞Εͨσʔλʹదͨ͠σʔλϨΠΞ΢τ —  ඇϦϨʔγϣφϧͳσʔλΛ૝ఆ͢ΔͨΊɺԿઍ΋ͷϑΟʔϧυΛ࣋ͭ ૄͳωετ͞ΕͨσʔλΛޮ཰తʹѻ͏͜ͱͷͰ͖ΔετϨʔδܗࣜͱ ΞϧΰϦζϜΛఏڙ͢Δ •  Ϧιʔεར༻ͷ࠷దԽ —  ಡΈࠐΈॻ͖ࠐΈϥΠϒϥϦʹ͓͍ͯɺ*0࡟ݮ΍Τϯίʔυσίʔυ ࣌ͷ$16ར༻ͷޮ཰Խ౳ͷ޻෉͕͞Ε͍ͯΔ •  ૬ޓӡ༻ੑ —  ༷ʑͳΫΤϦΤϯδϯ΍σʔλϚωδϝϯτπʔϧʹରԠͰ͖Δ σʔλϞσϧͷಛ௃

Slide 14

Slide 14 text

ωετ͞Εͨσʔλʹదͨ͠ σʔλϨΠΞ΢τ

Slide 15

Slide 15 text

ωετ͞Εͨσʔλͷ࣋ͪํ SFDPSEPSJFOUFE DPMVNOPSJFOUFE S S " # $ % & S S Ұ෦ΛಡΈग़ͨ͢Ίʹωετ͞Εͨ σʔλશͯΛಡΈग़͢ඞཁ͕͋Δ ඞཁͳσʔλʹͷΈΞΫηε͢Δ ͜ͱ͕Ͱ͖Δ

Slide 16

Slide 16 text

ωετ͞Εͨσʔλͷ࣋ͪํ SFDPSEPSJFOUFE DPMVNOPSJFOUFE S S " # $ % & S S ωετ͞Εͨσʔλ͢΂ͯΛ ಡΈग़͢ඞཁ͕͋Δ ඞཁͳσʔλͷΈΛಡΈग़͢ ͜ͱ͕Ͱ͖Δ σʔλߏ଄ͷ৘ใΛอ࣋ͭͭ͠αϒηοτ͝ͱʹ σʔλΛอଘ͍ͨ͠

Slide 17

Slide 17 text

%SFNFMͷσʔλϞσϧ ϑΟʔϧυ͝ͱʹɺ൓෮ϨϕϧͱఆٛϨϕϧΛ࣋ͭ͜ͱͰɺ σʔλߏ଄ͷ৘ใଛࣦͳ͠ʹσʔλΛอଘͰ͖Δ ˞൓෮ϨϕϧͱఆٛϨϕϧͷ಺༰͸ෳࡶͳͷͰ͜͜Ͱ͸આ໌Λলུ͠·͢ɻ ൓෮Ϩϕϧ ఆٛϨϕϧ Ҿ༻ɿIUUQTTUBUJDHPPHMFVTFSDPOUFOUDPNNFEJBSFTFBSDIHPPHMFDPNKBQVCTBSDIJWFQEG

Slide 18

Slide 18 text

Ϧιʔεར༻ͷ࠷దԽ

Slide 19

Slide 19 text

Ϧιʔεར༻ͷ࠷దԽํ๏ ̍ɽܕεϖγϑΟοΫͳΤϯίʔυํࣜ ̎ɽ$16ύΠϓϥΠϯͷޮ཰Խ ̏ɽσʔλϑΝΠϧͷϑΥʔϚοτ

Slide 20

Slide 20 text

Ϧιʔεར༻ͷ࠷దԽํ๏ ̍ɽܕεϖγϑΟοΫͳΤϯίʔυํࣜ ̎ɽ$16ύΠϓϥΠϯͷޮ཰Խ ̏ɽσʔλϑΝΠϧͷϑΥʔϚοτ

Slide 21

Slide 21 text

ܕεϖγϑΟοΫͳΤϯίʔυํࣜ •  1MBJO –  γϯϓϧʹ஋Λ࿈ଓͯ͠อଘ͢Δํ๏ •  %JDUJPOBSZ&ODPEJOH –  ࣙॻ৘ใΛ࡞੒ͯͦ͠ͷ৘ใΛ༻͍ͯΤϯίʔυ͢Δํ๏ –  ΧʔσΟφϦςΟ͕௿͍৔߹ʹ༗ޮ •  3VO-FOHUI&ODPEJOH –  ܁Γฦ͕͠ଓ͘஋Λޮ཰తʹอଘ͢Δํ๏ •  %FMUB&ODPEJOHT –  ιʔτ͞Εͨσʔλͷࠩ෼Λه࿥͢Δํ๏ –  5JNF4UBNQͳͲͷΠϯΫϦϝϯλϧͳσʔλͳͲ •  1SFpY&ODPEJOH –  %JDUJPOBSZ&ODPEJOH͕࢖͑ͳ͍ͱ͖ʹར༻͢Δ

Slide 22

Slide 22 text

%JDUJPOBSZ&ODPEJOH *% 'SVJUT "QQMF 0SBOHF -FNPO 8BUFSNFMPO .FMPO #BOBOB "QQMF 0SBOHF 0SBOHF -FNPO "QQMF 8BUFS .FMPO .FMPO #BOBOB "QQMF .FMPO &ODPEFE%BUB %JDUJPOBSZ

Slide 23

Slide 23 text

3VO-FOHUI&ODPEJOH B B B B B B C C C C B C &ODPEFE%BUB

Slide 24

Slide 24 text

%FMUB&ODPEJOH 100 101 101 102 101 101 102 101 99 100 105 107 114 116 119 120 121 101 101 102 101 101 102 101 99 100 105 107 114 116 119 120 121 CMPDL CMPDL SFGFSFODF 1 0 1 -1 0 1 -1 -2 1 5 2 7 2 3 1 1 ͭલͷ஋͔Βͷࠩ෼ 3 2 3 1 2 3 1 0 0 4 1 6 1 2 0 0 NJO EFMUB NJO EFMUB 11 10 11 01 10 11 01 00 000 100 001 110 001 010 000 000 Ϗοτ਺ Ϗοτ਺ CJUCJU CJUCJUCJU σʔλྔ͕ଟ͘ͳΔͱߋʹѹॖ཰͸্͕Δ

Slide 25

Slide 25 text

Ϧιʔεར༻ͷ࠷దԽํ๏ ̍ɽܕεϖγϑΟοΫͳΤϯίʔυํࣜ ̎ɽ$16ύΠϓϥΠϯͷޮ཰Խ ̏ɽσʔλϑΝΠϧͷϑΥʔϚοτ

Slide 26

Slide 26 text

$16ύΠϓϥΠϯ •  ܭࢉϦιʔεΛޮ཰తʹར༻Ͱ͖͍ͯΔঢ়ଶ •  ܭࢉϦιʔε͕ޮ཰తʹར༻Ͱ͖͍ͯͳ͍ঢ়ଶ 5JNF 5JNF 1*1&-*/& 1*1&-*/& #VCCMF

Slide 27

Slide 27 text

1BSRVFUʹ͓͚Δ$16ޮ཰Խ •  $16ύΠϓϥΠϯʹ͓͚Δ#VCCMFͷݪҼʹͳΓ΍͍͢ ෼ذ໋ྩʢJGจ౳ʣΛۃྗݮΒ͢ Ҿ༻ɿIUUQTXXXTMJEFTIBSFOFUDMPVEFSBIBEPPQTVNNJU SFGIUUQTQBSRVFUBQBDIFPSHQSFTFOUBUJPOT #JOBSZQBDLJOHͷྫ

Slide 28

Slide 28 text

Ϧιʔεར༻ͷ࠷దԽํ๏ ̍ɽܕεϖγϑΟοΫͳΤϯίʔυํࣜ ̎ɽ$16ύΠϓϥΠϯͷޮ཰Խ ̏ɽσʔλϑΝΠϧͷϑΥʔϚοτ

Slide 29

Slide 29 text

1BSRVFUͷϑΝΠϧϑΥʔϚοτ ฒྻॲཧɿ'JMF3PX(SPVQ *0ɿ$PMVNO$IVOL &ODPEJOH$PNQSFTTJPOɿ1BHF ͷ୯ҐͰॲཧΛฒྻԽ͠΍͍͢σʔλ ഑ஔ Ҿ༻ɿIUUQTQBSRVFUBQBDIFPSHEPDVNFOUBUJPOMBUFTU

Slide 30

Slide 30 text

1BSRVFUͷσʔλϨΠΞ΢τ NFUBEBUBϖʔδ͝ͱɺྻ͝ͱɺߦ͝ͱɺσʔλશମͱ͍ͬͨ୯ҐͰɺ εΩʔϚ৘ใϨίʔυ਺֤ྻͷ࠷খ஋ɾ࠷େ஋ͱ͍ͬͨ౷ܭ৘ใΛ ࣋ͭ ϝλσʔλͷ৘ใʹΑΓɺෆඞཁͳσʔλ͸ಡΈඈ͹͕͠Մೳ Ҿ༻ɿIUUQTQBSRVFUBQBDIFPSHEPDVNFOUBUJPOMBUFTU

Slide 31

Slide 31 text

ΦϓςΟϚΠβʹΑΔ࠷దԽ 4&-&$5DPM'30.NZUBCMFXIFSFDPM col1 col2 col3 a1 0 c1 a2 10 c2 a3 10 c3 a4 30 c4 a5 50 c5 •  QSPKFDUJPOQVTIEPXO ඞཁͳྻͷΈΛऔΓग़͢ •  QSFEJDBUFQVTIEPXO ඞཁͳߦͷΈΛऔΓग़͢ col1 col2 col3 a1 0 c1 a2 10 c2 a3 10 c3 a4 30 c4 a5 50 c5

Slide 32

Slide 32 text

4QBSLʹ͓͚ΔΦϓςΟϚΠβʹΑΔ࠷దԽͷྫ Ҿ༻ɿIUUQTEBUBCSJDLTDPNCMPHTQBSLTRMHSBEVBUFTGSPNBMQIBJOTQBSLIUNM

Slide 33

Slide 33 text

૬ޓӡ༻ੑ

Slide 34

Slide 34 text

1BSRVFUϓϩδΣΫτ $ʹΑΔ1BSRVFUϑΥʔϚοτͷ 3FBE8SJUFͷͨΊͷϥΠϒϥϦ 1BSRVFUͷσʔλϨΠΞ΢τͳͲͷ ఆٛͷϥΠϒϥϦ +BWBʹΑΔ1BSRVFUϑΥʔϚοτͷ 3FBE8SJUFͷͨΊͷϥΠϒϥϦ

Slide 35

Slide 35 text

ଞϓϩμΫτͱͷ૬ޓӡ༻ੑ Ҿ༻ɿIUUQTXXXTMJEFTIBSFOFUDMPVEFSBIBEPPQTVNNJU SFGIUUQTQBSRVFUBQBDIFPSHQSFTFOUBUJPOT

Slide 36

Slide 36 text

03$WT1BSRVFU

Slide 37

Slide 37 text

03$WT1BSRVFU •  ΞʔΩςΫνϟͷҧ͍   03$͸ωετ͞ΕͨσʔλΛͦΕͧΕผͷྻͱͯ͠ѻ͏   03$͸ϒϧʔϜϑΟϧλͷػೳ͕͋Δ   1BSRVFUʹ͸4DIFNBNFSHJOHͷػೳ͕͋Δ •  ϓϩμΫτ͝ͱʹɺ֤ϑΥʔϚοτ΁ͷରԠͷҧ͍ ʢϑΥʔϚοτ͸͋͘·Ͱ࠷దԽ͠΍͍͢σʔλܗࣜͰ͋ͬͯɺ࣮ࡍʹ ॲཧΛϓϩμΫτଆ͕ରԠ͍ͯ͠ͳ͍ͱҙຯ͕ͳ͍ʣ •  σʔλͷಛੑ΍ར༻͢ΔϓϩμΫτɺ࣮ߦ͢ΔΫΤϦͷύλʔϯʹΑͬͯɺ ࢖͍෼͚Δͷ͕େ੾ •  ৔߹ʹΑͬͯ͸03$΍1BSRVFUΑΓ΋4FRVFODF'JMFͳͲͷ΄͏͕ εϧʔϓοτ͕ྑ͍৔߹΋͋Δ •  Ұ൪͍͍ͷ͸ɺݕূͰεϧʔϓοτ΍σʔλαΠζΛ֬ೝ͢Δ͜ͱ

Slide 38

Slide 38 text

·ͱΊ •  ෼ੳ༻్ͷϏοάσʔλج൫ʹ͸ɺҰൠతʹΧϥϜφ ϑΥʔϚοτ͕ద͍ͯ͠Δ •  1BSRVFU͸ωετ͞Εͨσʔλ΁ͷରԠɺϦιʔεར༻ ͷ࠷దԽɺ૬ޓӡ༻ੑʹؔͯ͠޻෉͞Ε͍ͯΔ •  ϑΥʔϚοτ͸σʔλͷಛੑ΍ར༻͢ΔϓϩμΫτɺ࣮ ߦ͢ΔΫΤϦͷ಺༰ʹΑͬͯ࢖͍෼͚Δ

Slide 39

Slide 39 text

8F"SF)JSJOHʂ ؾܰʹΦϑΟεʹ༡ͼʹདྷ͍ͯͩ͘͞ʂ ͋ͱ3FUUZ͔͍ͭͬͯͩ͘͞ʂ

Slide 40

Slide 40 text

ࢀߟࢿྉ •  %SFNFM࿦จ –  IUUQTTUBUJDHPPHMFVTFSDPOUFOUDPNNFEJBSFTFBSDIHPPHMFDPNKBQVCT BSDIJWFQEG •  "QBDIF1BSRVFUυΩϡϝϯτ –  IUUQTQBSRVFUBQBDIFPSH •  (JU)VC –  IUUQTHJUIVCDPN1BSRVFU –  IUUQTHJUIVCDPNBQBDIFPSD •  "QBDIF03$υΩϡϝϯτ –  IUUQTPSDBQBDIFPSH •  &GpDJFOU%BUB4UPSBHFGPS"OBMZUJDTXJUI"QBDIF1BSRVFU –  IUUQTXXXTMJEFTIBSFOFUDMPVEFSBIBEPPQTVNNJU SFGIUUQT QBSRVFUBQBDIFPSHQSFTFOUBUJPOT •  +*3" –  IUUQTJTTVFTBQBDIFPSHKJSBJTTVFT