機械学習とデータ分析を支えるマルチクラウドなアーキテクチャの紹介/Multi Cloud Architecture Supporting Machine Learning and Data Analysis

Multi Cloud Architecture Supporting Machine Learning and Data Analysis in iQON.
at Minami Aoyama Night #1

Transcript

1. Multi-Cloud Architecture Supporting Machine Learning and Data Analysis ػցֶशͱσʔλ෼ੳΛࢧ͑ΔϚϧνΫϥ΢υΞʔΩςΫνϟͷ঺հ VASILY,Inc. @kyuns 2016.12.9 / Minami Aoyama Night #1
2. ࠓଜ
   խ޾ @kyuns / Ωϡϯ VASILY,Inc औక໾CTO / Co-Founder ೥
   :BIPP+"1"/ʹೖࣾɺ:BIPP'"4)*0/΍
   9#3"/%ͳͲͷαʔϏεͷ্ཱͪ͛ͷ։ൃΛ୲౰ɻ
   ϦίϝϯσʔγϣϯͷಛڐͳͲΛऔಘޙɺ
   ೥ʹಠཱɺ7"4*-:Λ૑ۀɻऔక໾$50ʹब೚
3. ઃཱ
   ೥݄
   ॴࡏ஍
   ౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ
   '
   ैۀһ
   ໊
   ࢿຊۚ
   ԯ
 ୅දऔక໾
   ۚࢁ
   ༟थ
 औక໾ࠓଜ
   խ޾
   
   ઍ༿
   େี
   4ZO
   άϧʔϓ
 ϑΝογϣϯʷςΫϊϩδʔ
   ΛڧΈͱͨ͠ձࣾ
4. ೔ຊதͷϑΝογϣϯ
   ECαΠτͷ঎඼σʔλΛܝࡌ ೔ຊ࠷େڃͷঁͷࢠͷͨΊͷ
   ϑΝογϣϯΞϓϦʮΞΠίϯʯ ձһ਺ 250ສਓ Google Play ϕετɾΞϓϦ 300Ҏ্ͷ&$αΠτ͔Β
   ສ఺ͷΞΠςϜܝࡌ
5. ೥࿈ଓϕετΞϓϦड৆
   
   (PPHMF
   1MBZ
   ϕετΠϊϕʔςΟϒΞϓϦ
   େ৆ड৆
6. ػցֶशͱσʔλ෼ੳΛࢧ͑Δ
   ϚϧνΫϥ΢υΞʔΩςΫνϟʔͷ঺հ ຊ೔ͷΞδΣϯμ ֤Ϋϥ΢υͷಛੑΛ׆͔ͯ͠
 ༻్ผʹ࠷దͳ΋ͷΛར༻͢Δ
7. ฐࣾͷσʔλΛ׆༻ͨ͠࠷ۙͷऔΓ૊Έ
8. 1.σʔλ෼ੳ ΞϓϦ಺ߦಈϩά΍Ϣʔβʔͷσʔλͷ෼ੳ
   Λ௒͑Δ7JFXͰͦΕͧΕͷࢪࡦͷ,1*Λ֬ೝ
9. 2.ػցֶश ສਓ͇ສ఺ͷΞΠςϜ͔Β
   Ϣʔβʔͷ޷ΈΛֶशͯ͠ΞΠςϜΛਪન͢ΔϨίϝϯυΤϯδϯ
10. 2.ػցֶशʢσΟʔϓϥʔχϯάʣ ը૾͔ΒΞΠςϜݕग़ྨࣅը૾ݕࡧ
    IUUQUFDIWBTJMZKQFOUSZEFUFDUJPO@BOE@SFUSJFWBM
11. 2.ػցֶशʢσΟʔϓϥʔχϯάʣ 7"&ͱ("/Λར༻ͨ͠ϑΝογϣϯΞΠςϜݕࡧ *#*4
    IUUQUFDIWBTJMZKQFOUSZSFUSJFWBM@VTJOH@WBF@HBO
12. ϚϧνΫϥ΢υ αʔϏεͷϝΠϯͱͳΔΠϯϑϥ σʔλ෼ੳج൫ͱͯ͠ͷΠϯϑϥ
13. σʔλ෼ੳ
14. RDS/Aurora Log Aggregater GoogleDataProc MongoDB GoogleDrive (SpreadSheet) API Server Web Server fluentd fluentd fluentd iQONͷϩάσʔλ·ΘΓͷΞʔΩςΫνϟ iOS/Android Puree AWS GCP BigQuery ΦϯϓϨ Tesla K Series / GTX1080
15. σʔλ෼ੳʹඞཁͳϩάऩू ΞϓϦ
    "OESPJE͸1VSFFͱJ04͸ಠࣗͷ4%,Λ௨ͯ͠ߦಈϩάΛऔಘ
    αʔόʔ
    Πϕϯτϩά͸શͯqVFOUEܦ༝Ͱ#JH2VFSZʹ4USFBNJOH
    *OTFSU
    8FC΍"1*ͷ/HJOYͷϩά΋શͯ4USFBNJOH
    *OTFSU
    Ϛελσʔλ
    3%4 .POHP%#ʹೖ͍ͬͯΔσʔλ΋͢΂ͯ#JH2VFSZʹίϐʔ IUUQTHJUIVCDPNDPPLQBEQVSFFBOESPJE
16. (PPHMF͕ఏڙ͢ΔϑϧϚωʔδυ%8)
 ѹ౗తʹ
    ؆୯
    ଎͍
    ͍҆ J20/Ͱ͸ҎԼͷΑ͏ͳσʔλશͯΛอଘ
    w ΞϓϦ1$εϚϗͷߦಈϩά
    w Ϛελʔ%#σʔλ GSPN
    3%4
    w Ϋϩʔϥʔͷશͯͷϩά
    w ϦίϝϯυΤϯδϯͷܭࢉ݁Ռ
    w σΟʔϓϥʔχϯά༻σʔλ BigQuery
17. ೔ςʔϒϧ͝ͱʹશͯͷϩάΛอଘ IUUQUFDIWBTJMZKQFOUSZCJHRVFSZ@EBUB@QMBUGPSN ྫ:ΞϓϦߦಈϩά
18. ࠷ۙͷBigQueryͷΞοϓσʔτ w 4UBOEBSE42-ରԠ
    w */4&3561%"5&%&-&5&΋Մೳʹ
    w Ұ࣌7JFX഑ྻߏ଄ମαϒΫΤϦͷॻ͚Δ৔ॴ ͕૿͑ͨɺͳͲ༷ʑͳػೳ͕ར༻Մೳʹͳͬͨ
    w ଞͷػೳมߋͳͲ͸ҎԼʹ·ͱΊͯ͋Γ·͢
 IUUQUFDIWBTJMZKQFOUSZCJHRVFSZTUBOEBSETRM
19. ؾʹͳΔྉۚ w ετϨʔδ
    (#
 ετϦʔϛϯάΠϯαʔτ
    (#
    w ΫΤϦྔ
    
    5#
 
 ஫ҙࣄ߲
 ೥݄͔Β͸ΫΤϦͷ಺༰ʹΑͬͯ஋ஈ͕มΘΔ
 NBYJNVN#JMMJOH5JFSΛઃఆਃ੥͢Δඞཁ͕͋Δ
    ࢀߟJ20/ͷ݄ؒར༻ঢ়گ ೥݄ 
 ෼ੳΫΤϦྔ
    5JC 5#
    
    
20. σʔλ෼ੳ͸Tableau #JH2VFSZ΍3%4΋σʔλιʔεͱͯ͠ϦΞϧλΠϜʹѻ͑Δ
 ΠϯλϥΫςΟϒͳ෼ੳ΍μογϡϘʔυͷ࡞੒͕Մೳ
    ࡉ͔͍σʔλ෼ੳ͸5BCMFBV%FTLUPQ
    ࣾ಺Ͱͷσʔλڞ༗ʹ͸5BCMFBV4FSWFS
21. μογϡϘʔυྫ:ΫϩʔϥʔՔಇ཰
22. ෼ੳྫ: ΰδϥ(※λϒϩʔͷαϯϓϧͰ͢)
23. ཚཱ͢ΔBIπʔϧ ҰମͲΕΛ࢖͑͹͍͍ͷ͔ʜʁ
24. BIπʔϧൺֱ ྉۚ #JH2VFSZ ୳ࡧత෼ੳ ߴػೳ ར఺ ܽ఺ 5BCMFBV
    %FTLUPQ
    
    4FSWFS ༗ྉ ˓ ˓ ˓ େ͖ͳσʔλ΋ಡΈ ࠐΊΔ ߴ͍ %BUB4UVEJP ແྉ ˓ ✕ ✕ ͍҆ ࡞ΕΔϘʔυ͸ͭ· Ͱ FYQMPSBUPSZ ༗ྉ ˚ ˓ ˓ LNFBOT47%͕6*Ͱ Ͱ͖Δ όά͕ଟ͍ɺେ͖͍ σʔλಡΈࠐΊͳ͍ SFEBTI ແྉ ˓ ✕ ˚ ͍҆ όά͕ଟ͍ 2VJDL4JHIU ༗ྉ ✕ ✕ ˚ "84ͱͷ਌࿨ੑ #2ະରԠ 4VQFSTFU ແྉ ✕ ✕ ˚ ͍҆ #2ະରԠ 1PXFS#* ༗ྉ ✕ ˓ ˓ .4ͱͷ਌࿨ੑ #2ະରԠ %PNP ༗ྉ ˓ ˓ ˓ ೔ຊͷେखۀͷಋೖ ࣄྫଟ਺ ߴ͍ ࡉ͔͍ൺֱ͸ޙ೔هࣄʹͯ঺հ༧ఆ
25. ݸਓతBIπʔϧબͼํͷίπ ࣾ಺ʹ#*ʹৄ͍͠ਓ͕͍Δ͓ۚΛଟগ͔͚ͯ΋͍͍
    ˠ
    5BCMFBV
    %FTLUPQ
    
    5BCMFBV
    4FSWFS
    ˞σʔλ෼ੳ͸ࢪࡦΛ൑அ͢ΔͷʹඇৗʹॏཁͳͷͰɺπʔϧࣗମʹόά͕͋Δ ͜ͱࣗମΫϦςΟΧϧ
    ͱΓ͓͋͑ͣۚΛ͔͚ͣʹ͍͍ײ͡ʹϨϙʔτΛΈ͍ͨ
 ˠ3FEBTI
    PS
    (PPHMF
    %BUB
    4UVEJP
    
26. όονॲཧ "JS'MPX
    "JSCOC͕ࣾ։ൃͨ͠όονॲཧ࣮ߦϑϨʔϜϫʔΫ
    ෳ਺ͷδϣϒͷґଘؔ܎ͷղܾ΍ϦτϥΠॲཧͳͲΛ͏·͘΍ͬͯ͘ΕΔ https://github.com/airbnb/airflow -VJHJ
    4QPUJGZ͕ࣾ։ൃͨ͠όονॲཧ࣮ߦϑϨʔϜϫʔΫ
    "JSqPXΑΓ΋γϯϓϧ https://github.com/spotify/luigi
27. ػցֶश
28. ػցֶश·ΘΓͷΞʔΩςΫνϟ جຊํ਑
    ɾ"84ͰͰ͖Δ͜ͱ͸"84Ͱ΍Δ
    ɾ#JH2VFSZʹؔ࿈͢Δ෦෼1ZUIPOͷ෦෼͸($1
    ɾ(16ͳͲΫϥ΢υͷ஋ஈ͕ߴͯ͘ɺݕূதʹϦιʔεΛৗ࣌େ ྔফඅ͢Δ΋ͷ͸ΦϯϓϨ
 ɾܭࢉࡁ݁Ռ͸3FEJTʹอଘͯ͠"1*͔Βݺͼग़͢
    ɾ֤Ϋϥ΢υؒͷ࣮૷͸#JH2VFSZΛத৺ͱͯ͠ૄ݁߹Λҙࣝ͢Δ
29. ྨࣅը૾APIͷΞʔΩςΫνϟ " 8 4 ( $ 1 Φ ϯ ϓ Ϩ ಛఆͷΞΠςϜʹྨࣅ͍ͯ͠ΔΞΠςϜ͕දࣔ͞ΕΔ ಛ௃ྔநग़εςοϓ ྨࣅ౓ܭࢉεςοϓ ΞΠςϜදࣔεςοϓ
30. ྨࣅը૾APIͷΞʔΩςΫνϟ "1*
    4FSWFS &$
 "1*
    
    3BJMT BigQuery Tesla / GTX1080 / Chainer 2.ը૾ΛDLɺಛ௃ྔநग़ GCP Batch / Python ֶश༻σʔλऔಘ " 8 4 ( $ 1 Φ ϯ ϓ Ϩ ը૾ͷಛ௃ྔΛอଘ ಛ௃ྔσʔλऔಘ ྨࣅ౓ܭࢉ ྨࣅը૾ܭࢉ݁Ռอଘ &$
    #BUDI
    
    3VCZ 3FEJT ܭࢉࡁΈσʔλऔಘ Ωϟογϡ࡞੒ ྨࣅը૾σʔλऔಘ ಛఆͷΞΠςϜʹྨࣅ͍ͯ͠ΔΞΠςϜ͕දࣔ͞ΕΔ
31. ϨίϝϯυΤϯδϯͷΞʔΩςΫνϟ(όονॲཧ൛) &$
 "1*
    
    3BJMT BigQuery GCP Batch / Python ߦಈσʔλऔಘ ਪન಺༰ܭࢉ ਪન݁Ռอଘ &$
    #BUDI
    
    3VCZ 3FEJT ਪન݁Ռऔಘ Ωϟογϡ࡞੒ ਪન݁Ռऔಘ " 8 4 ( $ 1 ೔ճɺϢʔβʔͷᅂ޷ʹ͋ͬͨΞΠςϜ͕දࣔ͞ΕΔ
32. ϨίϝϯυΤϯδϯͷΞʔΩςΫνϟ(ΦϯϥΠϯॲཧ൛) &$
 "1*
    
    3BJMT BigQuery GCP Batch / Python " 8 4 ( $ 1 ߦಈσʔλऔಘ ਪન಺༰ܭࢉ ਪન݁Ռอଘ GCP ߋ৽API / Flask ਪન݁Ռܭࢉˍߋ৽ ߦಈ݁Ռૹ৴
    3&45 ਪન݁Ռऔಘ ਪન݁Ռऔಘ
    3&45 Ϣʔβʔ͕ΞΠςϜΛݟΔͨͼʹΦεεϝΞΠςϜ͕มΘͬͯදࣔ͞ΕΔ 3FEJT
33. ·ͱΊ w σʔλ෼ੳ
 #JH2VFSZ 5BCMFBVͷ૊Έ߹Θ͕ͤͱͯ΋ྑ͍
    w ػցֶश
 ಛ௃ྔநग़ͳͲͷॏ͍ॲཧͷաఔ͸ΦϯϓϨͷ(16͕ Φεεϝ
 ͋ͱ͸($1
    "84ͷಛੑΛ׆͔ͯ͠ΞʔΩςΫνϟΛ ߟ͑Δ
 #JH2VFSZΛத৺ʹ͢Δͱઃܭ͠΍ͯ͘͢ӡ༻͠΍͍͢
34. We’re Hiring! ͋Γ͕ͱ͏͍͟͝·ͨ͠
    7"4*-:Ͱ͸
    ϑΝογϣϯʷςΫϊϩδʔʹ
    ڵຯͷ͋ΔΤϯδχΞΛืू͍ͯ͠·͢
    JOGP!WBTJMZKQ