機械学習とデータ分析を支えるマルチクラウドなアーキテクチャの紹介/Multi Cloud Architecture Supporting Machine Learning and Data Analysis

Multi Cloud Architecture Supporting Machine Learning and Data Analysis in iQON.
at Minami Aoyama Night #1

Transcript

1. Multi-Cloud
   Architecture
   Supporting
   Machine Learning
   and Data Analysis
   ػցֶशͱσʔλ෼ੳΛࢧ͑ΔϚϧνΫϥ΢υΞʔΩςΫνϟͷ঺հ
   VASILY,Inc. @kyuns
   2016.12.9 / Minami Aoyama Night #1
   

   View Slide

2. ࠓଜխ޾
   @kyuns / Ωϡϯ
   VASILY,Inc
   औక໾CTO / Co-Founder
   ೥:BIPP+"1"/ʹೖࣾɺ:BIPP'"4)*0/΍
   9#3"/%ͳͲͷαʔϏεͷ্ཱͪ͛ͷ։ൃΛ୲౰ɻ
   ϦίϝϯσʔγϣϯͷಛڐͳͲΛऔಘޙɺ
   ೥ʹಠཱɺ7"4*-:Λ૑ۀɻऔక໾$50ʹब೚
   

   View Slide

3. ઃཱ೥݄
   ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ'
   ैۀһ໊
   ࢿຊۚԯ

   ୅දऔక໾ۚࢁ༟थ

   औక໾ࠓଜխ޾ઍ༿େี
   4ZOάϧʔϓ

   ϑΝογϣϯʷςΫϊϩδʔ
   ΛڧΈͱͨ͠ձࣾ
   

   View Slide

4. ೔ຊதͷϑΝογϣϯ
   ECαΠτͷ঎඼σʔλΛܝࡌ
   ೔ຊ࠷େڃͷঁͷࢠͷͨΊͷ
   ϑΝογϣϯΞϓϦʮΞΠίϯʯ
   ձһ਺
   250ສਓ
   Google Play
   ϕετɾΞϓϦ
   300Ҏ্ͷ&$αΠτ͔Β
   ສ఺ͷΞΠςϜܝࡌ
   

   View Slide

5. ೥࿈ଓϕετΞϓϦड৆
   (PPHMF1MBZϕετΠϊϕʔςΟϒΞϓϦେ৆ड৆
   

   View Slide

6. ػցֶशͱσʔλ෼ੳΛࢧ͑Δ
   ϚϧνΫϥ΢υΞʔΩςΫνϟʔͷ঺հ
   ຊ೔ͷΞδΣϯμ
   ֤Ϋϥ΢υͷಛੑΛ׆͔ͯ͠

   ༻్ผʹ࠷దͳ΋ͷΛར༻͢Δ
   

   View Slide

7. ฐࣾͷσʔλΛ׆༻ͨ͠࠷ۙͷऔΓ૊Έ
   

   View Slide

8. 1.σʔλ෼ੳ
   ΞϓϦ಺ߦಈϩά΍Ϣʔβʔͷσʔλͷ෼ੳ
   Λ௒͑Δ7JFXͰͦΕͧΕͷࢪࡦͷ,1*Λ֬ೝ
   

   View Slide

9. 2.ػցֶश
   ສਓ͇ສ఺ͷΞΠςϜ͔Β
   Ϣʔβʔͷ޷ΈΛֶशͯ͠ΞΠςϜΛਪન͢ΔϨίϝϯυΤϯδϯ
   

   View Slide

10. 2.ػցֶशʢσΟʔϓϥʔχϯάʣ
    ը૾͔ΒΞΠςϜݕग़ྨࣅը૾ݕࡧ
    IUUQUFDIWBTJMZKQFOUSZEFUFDUJPO@BOE@SFUSJFWBM
    

    View Slide

11. 2.ػցֶशʢσΟʔϓϥʔχϯάʣ
    7"&ͱ("/Λར༻ͨ͠ϑΝογϣϯΞΠςϜݕࡧ *#*4
    
    IUUQUFDIWBTJMZKQFOUSZSFUSJFWBM@VTJOH@WBF@HBO
    

    View Slide

12. ϚϧνΫϥ΢υ
    αʔϏεͷϝΠϯͱͳΔΠϯϑϥ
    σʔλ෼ੳج൫ͱͯ͠ͷΠϯϑϥ
    

    View Slide

13. σʔλ෼ੳ
    

    View Slide

14. RDS/Aurora
    Log Aggregater
    GoogleDataProc
    MongoDB
    GoogleDrive
    (SpreadSheet)
    API Server Web Server
    fluentd fluentd
    fluentd
    iQONͷϩάσʔλ·ΘΓͷΞʔΩςΫνϟ
    iOS/Android
    Puree
    AWS
    GCP
    BigQuery
    ΦϯϓϨ
    Tesla K Series / GTX1080
    

    View Slide

15. σʔλ෼ੳʹඞཁͳϩάऩू
    ΞϓϦ
    "OESPJE͸1VSFFͱJ04͸ಠࣗͷ4%,Λ௨ͯ͠ߦಈϩάΛऔಘ
    αʔόʔ
    Πϕϯτϩά͸શͯqVFOUEܦ༝Ͱ#JH2VFSZʹ4USFBNJOH*OTFSU
    8FC΍"1*ͷ/HJOYͷϩά΋શͯ4USFBNJOH*OTFSU
    Ϛελσʔλ
    3%4 .POHP%#ʹೖ͍ͬͯΔσʔλ΋͢΂ͯ#JH2VFSZʹίϐʔ
    IUUQTHJUIVCDPNDPPLQBEQVSFFBOESPJE
    

    View Slide

16. (PPHMF͕ఏڙ͢ΔϑϧϚωʔδυ%8)

    ѹ౗తʹ؆୯଎͍͍҆
    J20/Ͱ͸ҎԼͷΑ͏ͳσʔλશͯΛอଘ
    w ΞϓϦ1$εϚϗͷߦಈϩά
    w Ϛελʔ%#σʔλ GSPN3%4
    
    w Ϋϩʔϥʔͷશͯͷϩά
    w ϦίϝϯυΤϯδϯͷܭࢉ݁Ռ
    w σΟʔϓϥʔχϯά༻σʔλ
    BigQuery
    

    View Slide

17. ೔ςʔϒϧ͝ͱʹશͯͷϩάΛอଘ
    IUUQUFDIWBTJMZKQFOUSZCJHRVFSZ@EBUB@QMBUGPSN
    ྫ:ΞϓϦߦಈϩά
    

    View Slide

18. ࠷ۙͷBigQueryͷΞοϓσʔτ
    w 4UBOEBSE42-ରԠ
    w */4&3561%"5&%&-&5&΋Մೳʹ
    w Ұ࣌7JFX഑ྻߏ଄ମαϒΫΤϦͷॻ͚Δ৔ॴ
    ͕૿͑ͨɺͳͲ༷ʑͳػೳ͕ར༻Մೳʹͳͬͨ
    w ଞͷػೳมߋͳͲ͸ҎԼʹ·ͱΊͯ͋Γ·͢

    IUUQUFDIWBTJMZKQFOUSZCJHRVFSZTUBOEBSETRM
    

    View Slide

19. ؾʹͳΔྉۚ
    w ετϨʔδ(#

    ετϦʔϛϯάΠϯαʔτ(#
    w ΫΤϦྔ5#

    

    ஫ҙࣄ߲

    ೥݄͔Β͸ΫΤϦͷ಺༰ʹΑͬͯ஋ஈ͕มΘΔ

    NBYJNVN#JMMJOH5JFSΛઃఆਃ੥͢Δඞཁ͕͋Δ
    ࢀߟJ20/ͷ݄ؒར༻ঢ়گ ೥݄
    

    ෼ੳΫΤϦྔ5JC 5#
    
    

    View Slide

20. σʔλ෼ੳ͸Tableau
    #JH2VFSZ΍3%4΋σʔλιʔεͱͯ͠ϦΞϧλΠϜʹѻ͑Δ

    ΠϯλϥΫςΟϒͳ෼ੳ΍μογϡϘʔυͷ࡞੒͕Մೳ
    ࡉ͔͍σʔλ෼ੳ͸5BCMFBV%FTLUPQ
    ࣾ಺Ͱͷσʔλڞ༗ʹ͸5BCMFBV4FSWFS
    

    View Slide

21. μογϡϘʔυྫ:ΫϩʔϥʔՔಇ཰
    

    View Slide

22. ෼ੳྫ: ΰδϥ(※λϒϩʔͷαϯϓϧͰ͢)
    

    View Slide

23. ཚཱ͢ΔBIπʔϧ
    ҰମͲΕΛ࢖͑͹͍͍ͷ͔ʜʁ
    

    View Slide

24. BIπʔϧൺֱ
    ྉۚ #JH2VFSZ ୳ࡧత෼ੳ ߴػೳ ར఺ ܽ఺
    5BCMFBV%FTLUPQ
    4FSWFS
    ༗ྉ ˓ ˓ ˓ େ͖ͳσʔλ΋ಡΈ
    ࠐΊΔ
    ߴ͍
    %BUB4UVEJP ແྉ ˓ ✕ ✕ ͍҆
    ࡞ΕΔϘʔυ͸ͭ·
    Ͱ
    FYQMPSBUPSZ ༗ྉ ˚ ˓ ˓ LNFBOT47%͕6*Ͱ
    Ͱ͖Δ
    όά͕ଟ͍ɺେ͖͍
    σʔλಡΈࠐΊͳ͍
    SFEBTI ແྉ ˓ ✕ ˚ ͍҆ όά͕ଟ͍
    2VJDL4JHIU ༗ྉ ✕ ✕ ˚ "84ͱͷ਌࿨ੑ #2ະରԠ
    4VQFSTFU ແྉ ✕ ✕ ˚ ͍҆ #2ະରԠ
    1PXFS#* ༗ྉ ✕ ˓ ˓ .4ͱͷ਌࿨ੑ #2ະରԠ
    %PNP ༗ྉ ˓ ˓ ˓ ೔ຊͷେखۀͷಋೖ
    ࣄྫଟ਺
    ߴ͍
    ࡉ͔͍ൺֱ͸ޙ೔هࣄʹͯ঺հ༧ఆ
    

    View Slide

25. ݸਓతBIπʔϧબͼํͷίπ
    ࣾ಺ʹ#*ʹৄ͍͠ਓ͕͍Δ͓ۚΛଟগ͔͚ͯ΋͍͍
    ˠ5BCMFBV%FTLUPQ5BCMFBV4FSWFS
    ˞σʔλ෼ੳ͸ࢪࡦΛ൑அ͢ΔͷʹඇৗʹॏཁͳͷͰɺπʔϧࣗମʹόά͕͋Δ
    ͜ͱࣗମΫϦςΟΧϧ
    ͱΓ͓͋͑ͣۚΛ͔͚ͣʹ͍͍ײ͡ʹϨϙʔτΛΈ͍ͨ

    ˠ3FEBTIPS(PPHMF%BUB4UVEJP
    

    View Slide

26. όονॲཧ
    "JS'MPX
    "JSCOC͕ࣾ։ൃͨ͠όονॲཧ࣮ߦϑϨʔϜϫʔΫ
    ෳ਺ͷδϣϒͷґଘؔ܎ͷղܾ΍ϦτϥΠॲཧͳͲΛ͏·͘΍ͬͯ͘ΕΔ
    https://github.com/airbnb/airflow
    -VJHJ
    4QPUJGZ͕ࣾ։ൃͨ͠όονॲཧ࣮ߦϑϨʔϜϫʔΫ
    "JSqPXΑΓ΋γϯϓϧ
    https://github.com/spotify/luigi
    

    View Slide

27. ػցֶश
    

    View Slide

28. ػցֶश·ΘΓͷΞʔΩςΫνϟ
    جຊํ਑
    ɾ"84ͰͰ͖Δ͜ͱ͸"84Ͱ΍Δ
    ɾ#JH2VFSZʹؔ࿈͢Δ෦෼1ZUIPOͷ෦෼͸($1
    ɾ(16ͳͲΫϥ΢υͷ஋ஈ͕ߴͯ͘ɺݕূதʹϦιʔεΛৗ࣌େ
    ྔফඅ͢Δ΋ͷ͸ΦϯϓϨ

    ɾܭࢉࡁ݁Ռ͸3FEJTʹอଘͯ͠"1*͔Βݺͼग़͢
    ɾ֤Ϋϥ΢υؒͷ࣮૷͸#JH2VFSZΛத৺ͱͯ͠ૄ݁߹Λҙࣝ͢Δ
    

    View Slide

29. ྨࣅը૾APIͷΞʔΩςΫνϟ
    "
    8
    4
    (
    $
    1
    Φ
    ϯ
    ϓ
    Ϩ
    ಛఆͷΞΠςϜʹྨࣅ͍ͯ͠ΔΞΠςϜ͕දࣔ͞ΕΔ
    ಛ௃ྔநग़εςοϓ
    ྨࣅ౓ܭࢉεςοϓ
    ΞΠςϜදࣔεςοϓ
    

    View Slide

30. ྨࣅը૾APIͷΞʔΩςΫνϟ
    "1*4FSWFS
    &$

    "1*3BJMT
    BigQuery
    Tesla / GTX1080 / Chainer
    2.ը૾ΛDLɺಛ௃ྔநग़
    GCP
    Batch / Python
    ֶश༻σʔλऔಘ
    "
    8
    4
    (
    $
    1
    Φ
    ϯ
    ϓ
    Ϩ
    ը૾ͷಛ௃ྔΛอଘ
    ಛ௃ྔσʔλऔಘ
    ྨࣅ౓ܭࢉ
    ྨࣅը૾ܭࢉ݁Ռอଘ
    &$
    #BUDI3VCZ
    3FEJT
    ܭࢉࡁΈσʔλऔಘ
    Ωϟογϡ࡞੒
    ྨࣅը૾σʔλऔಘ
    ಛఆͷΞΠςϜʹྨࣅ͍ͯ͠ΔΞΠςϜ͕දࣔ͞ΕΔ
    

    View Slide

31. ϨίϝϯυΤϯδϯͷΞʔΩςΫνϟ(όονॲཧ൛)
    &$

    "1*3BJMT
    BigQuery
    GCP
    Batch / Python
    ߦಈσʔλऔಘ
    ਪન಺༰ܭࢉ
    ਪન݁Ռอଘ
    &$
    #BUDI3VCZ
    3FEJT
    ਪન݁Ռऔಘ
    Ωϟογϡ࡞੒
    ਪન݁Ռऔಘ
    "
    8
    4
    (
    $
    1
    ೔ճɺϢʔβʔͷᅂ޷ʹ͋ͬͨΞΠςϜ͕දࣔ͞ΕΔ
    

    View Slide

32. ϨίϝϯυΤϯδϯͷΞʔΩςΫνϟ(ΦϯϥΠϯॲཧ൛)
    &$

    "1*3BJMT
    BigQuery
    GCP
    Batch / Python
    "
    8
    4
    (
    $
    1
    ߦಈσʔλऔಘ
    ਪન಺༰ܭࢉ
    ਪન݁Ռอଘ
    GCP
    ߋ৽API / Flask
    ਪન݁Ռܭࢉˍߋ৽
    ߦಈ݁Ռૹ৴
    3&45
    ਪન݁Ռऔಘ
    ਪન݁Ռऔಘ
    3&45
    Ϣʔβʔ͕ΞΠςϜΛݟΔͨͼʹΦεεϝΞΠςϜ͕มΘͬͯදࣔ͞ΕΔ
    3FEJT
    

    View Slide

33. ·ͱΊ
    w σʔλ෼ੳ

    #JH2VFSZ5BCMFBVͷ૊Έ߹Θ͕ͤͱͯ΋ྑ͍
    w ػցֶश

    ಛ௃ྔநग़ͳͲͷॏ͍ॲཧͷաఔ͸ΦϯϓϨͷ(16͕
    Φεεϝ

    ͋ͱ͸($1
    "84ͷಛੑΛ׆͔ͯ͠ΞʔΩςΫνϟΛ
    ߟ͑Δ

    #JH2VFSZΛத৺ʹ͢Δͱઃܭ͠΍ͯ͘͢ӡ༻͠΍͍͢
    

    View Slide

34. We’re Hiring!
    ͋Γ͕ͱ͏͍͟͝·ͨ͠
    7"4*-:Ͱ͸
    ϑΝογϣϯʷςΫϊϩδʔʹ
    ڵຯͷ͋ΔΤϯδχΞΛืू͍ͯ͠·͢
    JOGP!WBTJMZKQ
    

    View Slide