Upgrade to Pro — share decks privately, control downloads, hide ads and more …

機械学習とデータ分析を支えるマルチクラウドなアーキテクチャの紹介/Multi Cloud Architecture Supporting Machine Learning and Data Analysis

機械学習とデータ分析を支えるマルチクラウドなアーキテクチャの紹介/Multi Cloud Architecture Supporting Machine Learning and Data Analysis

Multi Cloud Architecture Supporting Machine Learning and Data Analysis in iQON.
at Minami Aoyama Night #1

Masayuki Imamura

December 09, 2016
Tweet

More Decks by Masayuki Imamura

Other Decks in Programming

Transcript

  1. Multi-Cloud
    Architecture
    Supporting
    Machine Learning
    and Data Analysis
    ػցֶशͱσʔλ෼ੳΛࢧ͑ΔϚϧνΫϥ΢υΞʔΩςΫνϟͷ঺հ
    VASILY,Inc. @kyuns
    2016.12.9 / Minami Aoyama Night #1

    View Slide

  2. ࠓଜխ޾
    @kyuns / Ωϡϯ
    VASILY,Inc
    औక໾CTO / Co-Founder
    ೥:BIPP+"1"/ʹೖࣾɺ:BIPP'"4)*0/΍
    9#3"/%ͳͲͷαʔϏεͷ্ཱͪ͛ͷ։ൃΛ୲౰ɻ
    ϦίϝϯσʔγϣϯͷಛڐͳͲΛऔಘޙɺ
    ೥ʹಠཱɺ7"4*-:Λ૑ۀɻऔక໾$50ʹब೚

    View Slide

  3. ઃཱ೥݄
    ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ'
    ैۀһ໊
    ࢿຊۚԯ

    ୅දऔక໾ۚࢁ༟थ

    औక໾ࠓଜխ޾ઍ༿େี
    4ZOάϧʔϓ

    ϑΝογϣϯʷςΫϊϩδʔ
    ΛڧΈͱͨ͠ձࣾ

    View Slide

  4. ೔ຊதͷϑΝογϣϯ
    ECαΠτͷ঎඼σʔλΛܝࡌ
    ೔ຊ࠷େڃͷঁͷࢠͷͨΊͷ
    ϑΝογϣϯΞϓϦʮΞΠίϯʯ
    ձһ਺
    250ສਓ
    Google Play
    ϕετɾΞϓϦ
    300Ҏ্ͷ&$αΠτ͔Β
    ສ఺ͷΞΠςϜܝࡌ

    View Slide

  5. ೥࿈ଓϕετΞϓϦड৆
    (PPHMF1MBZϕετΠϊϕʔςΟϒΞϓϦେ৆ड৆

    View Slide

  6. ػցֶशͱσʔλ෼ੳΛࢧ͑Δ
    ϚϧνΫϥ΢υΞʔΩςΫνϟʔͷ঺հ
    ຊ೔ͷΞδΣϯμ
    ֤Ϋϥ΢υͷಛੑΛ׆͔ͯ͠

    ༻్ผʹ࠷దͳ΋ͷΛར༻͢Δ

    View Slide

  7. ฐࣾͷσʔλΛ׆༻ͨ͠࠷ۙͷऔΓ૊Έ

    View Slide

  8. 1.σʔλ෼ੳ
    ΞϓϦ಺ߦಈϩά΍Ϣʔβʔͷσʔλͷ෼ੳ
    Λ௒͑Δ7JFXͰͦΕͧΕͷࢪࡦͷ,1*Λ֬ೝ

    View Slide

  9. 2.ػցֶश
    ສਓ͇ສ఺ͷΞΠςϜ͔Β
    Ϣʔβʔͷ޷ΈΛֶशͯ͠ΞΠςϜΛਪન͢ΔϨίϝϯυΤϯδϯ

    View Slide

  10. 2.ػցֶशʢσΟʔϓϥʔχϯάʣ
    ը૾͔ΒΞΠςϜݕग़ྨࣅը૾ݕࡧ
    IUUQUFDIWBTJMZKQFOUSZEFUFDUJPO@BOE@SFUSJFWBM

    View Slide

  11. 2.ػցֶशʢσΟʔϓϥʔχϯάʣ
    7"&ͱ("/Λར༻ͨ͠ϑΝογϣϯΞΠςϜݕࡧ *#*4

    IUUQUFDIWBTJMZKQFOUSZSFUSJFWBM@VTJOH@WBF@HBO

    View Slide

  12. ϚϧνΫϥ΢υ
    αʔϏεͷϝΠϯͱͳΔΠϯϑϥ
    σʔλ෼ੳج൫ͱͯ͠ͷΠϯϑϥ

    View Slide

  13. σʔλ෼ੳ

    View Slide

  14. RDS/Aurora
    Log Aggregater
    GoogleDataProc
    MongoDB
    GoogleDrive
    (SpreadSheet)
    API Server Web Server
    fluentd fluentd
    fluentd
    iQONͷϩάσʔλ·ΘΓͷΞʔΩςΫνϟ
    iOS/Android
    Puree
    AWS
    GCP
    BigQuery
    ΦϯϓϨ
    Tesla K Series / GTX1080

    View Slide

  15. σʔλ෼ੳʹඞཁͳϩάऩू
    ΞϓϦ
    "OESPJE͸1VSFFͱJ04͸ಠࣗͷ4%,Λ௨ͯ͠ߦಈϩάΛऔಘ
    αʔόʔ
    Πϕϯτϩά͸શͯqVFOUEܦ༝Ͱ#JH2VFSZʹ4USFBNJOH*OTFSU
    8FC΍"1*ͷ/HJOYͷϩά΋શͯ4USFBNJOH*OTFSU
    Ϛελσʔλ
    3%4 .POHP%#ʹೖ͍ͬͯΔσʔλ΋͢΂ͯ#JH2VFSZʹίϐʔ
    IUUQTHJUIVCDPNDPPLQBEQVSFFBOESPJE

    View Slide

  16. (PPHMF͕ఏڙ͢ΔϑϧϚωʔδυ%8)

    ѹ౗తʹ؆୯଎͍͍҆
    J20/Ͱ͸ҎԼͷΑ͏ͳσʔλશͯΛอଘ
    w ΞϓϦ1$εϚϗͷߦಈϩά
    w Ϛελʔ%#σʔλ GSPN3%4

    w Ϋϩʔϥʔͷશͯͷϩά
    w ϦίϝϯυΤϯδϯͷܭࢉ݁Ռ
    w σΟʔϓϥʔχϯά༻σʔλ
    BigQuery

    View Slide

  17. ೔ςʔϒϧ͝ͱʹશͯͷϩάΛอଘ
    IUUQUFDIWBTJMZKQFOUSZCJHRVFSZ@EBUB@QMBUGPSN
    ྫ:ΞϓϦߦಈϩά

    View Slide

  18. ࠷ۙͷBigQueryͷΞοϓσʔτ
    w 4UBOEBSE42-ରԠ
    w */4&3561%"5&%&-&5&΋Մೳʹ
    w Ұ࣌7JFX഑ྻߏ଄ମαϒΫΤϦͷॻ͚Δ৔ॴ
    ͕૿͑ͨɺͳͲ༷ʑͳػೳ͕ར༻Մೳʹͳͬͨ
    w ଞͷػೳมߋͳͲ͸ҎԼʹ·ͱΊͯ͋Γ·͢

    IUUQUFDIWBTJMZKQFOUSZCJHRVFSZTUBOEBSETRM

    View Slide

  19. ؾʹͳΔྉۚ
    w ετϨʔδ(#

    ετϦʔϛϯάΠϯαʔτ(#
    w ΫΤϦྔ5#


    ஫ҙࣄ߲

    ೥݄͔Β͸ΫΤϦͷ಺༰ʹΑͬͯ஋ஈ͕มΘΔ

    NBYJNVN#JMMJOH5JFSΛઃఆਃ੥͢Δඞཁ͕͋Δ
    ࢀߟJ20/ͷ݄ؒར༻ঢ়گ ೥݄

    ෼ੳΫΤϦྔ5JC 5#

    View Slide

  20. σʔλ෼ੳ͸Tableau
    #JH2VFSZ΍3%4΋σʔλιʔεͱͯ͠ϦΞϧλΠϜʹѻ͑Δ

    ΠϯλϥΫςΟϒͳ෼ੳ΍μογϡϘʔυͷ࡞੒͕Մೳ
    ࡉ͔͍σʔλ෼ੳ͸5BCMFBV%FTLUPQ
    ࣾ಺Ͱͷσʔλڞ༗ʹ͸5BCMFBV4FSWFS

    View Slide

  21. μογϡϘʔυྫ:ΫϩʔϥʔՔಇ཰

    View Slide

  22. ෼ੳྫ: ΰδϥ(※λϒϩʔͷαϯϓϧͰ͢)

    View Slide

  23. ཚཱ͢ΔBIπʔϧ
    ҰମͲΕΛ࢖͑͹͍͍ͷ͔ʜʁ

    View Slide

  24. BIπʔϧൺֱ
    ྉۚ #JH2VFSZ ୳ࡧత෼ੳ ߴػೳ ར఺ ܽ఺
    5BCMFBV%FTLUPQ
    4FSWFS
    ༗ྉ ˓ ˓ ˓ େ͖ͳσʔλ΋ಡΈ
    ࠐΊΔ
    ߴ͍
    %BUB4UVEJP ແྉ ˓ ✕ ✕ ͍҆
    ࡞ΕΔϘʔυ͸ͭ·
    Ͱ
    FYQMPSBUPSZ ༗ྉ ˚ ˓ ˓ LNFBOT47%͕6*Ͱ
    Ͱ͖Δ
    όά͕ଟ͍ɺେ͖͍
    σʔλಡΈࠐΊͳ͍
    SFEBTI ແྉ ˓ ✕ ˚ ͍҆ όά͕ଟ͍
    2VJDL4JHIU ༗ྉ ✕ ✕ ˚ "84ͱͷ਌࿨ੑ #2ະରԠ
    4VQFSTFU ແྉ ✕ ✕ ˚ ͍҆ #2ະରԠ
    1PXFS#* ༗ྉ ✕ ˓ ˓ .4ͱͷ਌࿨ੑ #2ະରԠ
    %PNP ༗ྉ ˓ ˓ ˓ ೔ຊͷେखۀͷಋೖ
    ࣄྫଟ਺
    ߴ͍
    ࡉ͔͍ൺֱ͸ޙ೔هࣄʹͯ঺հ༧ఆ

    View Slide

  25. ݸਓతBIπʔϧબͼํͷίπ
    ࣾ಺ʹ#*ʹৄ͍͠ਓ͕͍Δ͓ۚΛଟগ͔͚ͯ΋͍͍
    ˠ5BCMFBV%FTLUPQ5BCMFBV4FSWFS
    ˞σʔλ෼ੳ͸ࢪࡦΛ൑அ͢ΔͷʹඇৗʹॏཁͳͷͰɺπʔϧࣗମʹόά͕͋Δ
    ͜ͱࣗମΫϦςΟΧϧ
    ͱΓ͓͋͑ͣۚΛ͔͚ͣʹ͍͍ײ͡ʹϨϙʔτΛΈ͍ͨ

    ˠ3FEBTIPS(PPHMF%BUB4UVEJP

    View Slide

  26. όονॲཧ
    "JS'MPX
    "JSCOC͕ࣾ։ൃͨ͠όονॲཧ࣮ߦϑϨʔϜϫʔΫ
    ෳ਺ͷδϣϒͷґଘؔ܎ͷղܾ΍ϦτϥΠॲཧͳͲΛ͏·͘΍ͬͯ͘ΕΔ
    https://github.com/airbnb/airflow
    -VJHJ
    4QPUJGZ͕ࣾ։ൃͨ͠όονॲཧ࣮ߦϑϨʔϜϫʔΫ
    "JSqPXΑΓ΋γϯϓϧ
    https://github.com/spotify/luigi

    View Slide

  27. ػցֶश

    View Slide

  28. ػցֶश·ΘΓͷΞʔΩςΫνϟ
    جຊํ਑
    ɾ"84ͰͰ͖Δ͜ͱ͸"84Ͱ΍Δ
    ɾ#JH2VFSZʹؔ࿈͢Δ෦෼1ZUIPOͷ෦෼͸($1
    ɾ(16ͳͲΫϥ΢υͷ஋ஈ͕ߴͯ͘ɺݕূதʹϦιʔεΛৗ࣌େ
    ྔফඅ͢Δ΋ͷ͸ΦϯϓϨ

    ɾܭࢉࡁ݁Ռ͸3FEJTʹอଘͯ͠"1*͔Βݺͼग़͢
    ɾ֤Ϋϥ΢υؒͷ࣮૷͸#JH2VFSZΛத৺ͱͯ͠ૄ݁߹Λҙࣝ͢Δ

    View Slide

  29. ྨࣅը૾APIͷΞʔΩςΫνϟ
    "
    8
    4
    (
    $
    1
    Φ
    ϯ
    ϓ
    Ϩ
    ಛఆͷΞΠςϜʹྨࣅ͍ͯ͠ΔΞΠςϜ͕දࣔ͞ΕΔ
    ಛ௃ྔநग़εςοϓ
    ྨࣅ౓ܭࢉεςοϓ
    ΞΠςϜදࣔεςοϓ

    View Slide

  30. ྨࣅը૾APIͷΞʔΩςΫνϟ
    "1*4FSWFS
    &$

    "1*3BJMT
    BigQuery
    Tesla / GTX1080 / Chainer
    2.ը૾ΛDLɺಛ௃ྔநग़
    GCP
    Batch / Python
    ֶश༻σʔλऔಘ
    "
    8
    4
    (
    $
    1
    Φ
    ϯ
    ϓ
    Ϩ
    ը૾ͷಛ௃ྔΛอଘ
    ಛ௃ྔσʔλऔಘ
    ྨࣅ౓ܭࢉ
    ྨࣅը૾ܭࢉ݁Ռอଘ
    &$
    #BUDI3VCZ
    3FEJT
    ܭࢉࡁΈσʔλऔಘ
    Ωϟογϡ࡞੒
    ྨࣅը૾σʔλऔಘ
    ಛఆͷΞΠςϜʹྨࣅ͍ͯ͠ΔΞΠςϜ͕දࣔ͞ΕΔ

    View Slide

  31. ϨίϝϯυΤϯδϯͷΞʔΩςΫνϟ(όονॲཧ൛)
    &$

    "1*3BJMT
    BigQuery
    GCP
    Batch / Python
    ߦಈσʔλऔಘ
    ਪન಺༰ܭࢉ
    ਪન݁Ռอଘ
    &$
    #BUDI3VCZ
    3FEJT
    ਪન݁Ռऔಘ
    Ωϟογϡ࡞੒
    ਪન݁Ռऔಘ
    "
    8
    4
    (
    $
    1
    ೔ճɺϢʔβʔͷᅂ޷ʹ͋ͬͨΞΠςϜ͕දࣔ͞ΕΔ

    View Slide

  32. ϨίϝϯυΤϯδϯͷΞʔΩςΫνϟ(ΦϯϥΠϯॲཧ൛)
    &$

    "1*3BJMT
    BigQuery
    GCP
    Batch / Python
    "
    8
    4
    (
    $
    1
    ߦಈσʔλऔಘ
    ਪન಺༰ܭࢉ
    ਪન݁Ռอଘ
    GCP
    ߋ৽API / Flask
    ਪન݁Ռܭࢉˍߋ৽
    ߦಈ݁Ռૹ৴
    3&45
    ਪન݁Ռऔಘ
    ਪન݁Ռऔಘ
    3&45
    Ϣʔβʔ͕ΞΠςϜΛݟΔͨͼʹΦεεϝΞΠςϜ͕มΘͬͯදࣔ͞ΕΔ
    3FEJT

    View Slide

  33. ·ͱΊ
    w σʔλ෼ੳ

    #JH2VFSZ5BCMFBVͷ૊Έ߹Θ͕ͤͱͯ΋ྑ͍
    w ػցֶश

    ಛ௃ྔநग़ͳͲͷॏ͍ॲཧͷաఔ͸ΦϯϓϨͷ(16͕
    Φεεϝ

    ͋ͱ͸($1
    "84ͷಛੑΛ׆͔ͯ͠ΞʔΩςΫνϟΛ
    ߟ͑Δ

    #JH2VFSZΛத৺ʹ͢Δͱઃܭ͠΍ͯ͘͢ӡ༻͠΍͍͢

    View Slide

  34. We’re Hiring!
    ͋Γ͕ͱ͏͍͟͝·ͨ͠
    7"4*-:Ͱ͸
    ϑΝογϣϯʷςΫϊϩδʔʹ
    ڵຯͷ͋ΔΤϯδχΞΛืू͍ͯ͠·͢
    JOGP!WBTJMZKQ

    View Slide