Upgrade to Pro — share decks privately, control downloads, hide ads and more …

機械学習とデータ分析を支えるマルチクラウドなアーキテクチャの紹介/Multi Cloud Architecture Supporting Machine Learning and Data Analysis

機械学習とデータ分析を支えるマルチクラウドなアーキテクチャの紹介/Multi Cloud Architecture Supporting Machine Learning and Data Analysis

Multi Cloud Architecture Supporting Machine Learning and Data Analysis in iQON.
at Minami Aoyama Night #1

5b25474e615258ee6421d5b9ef2d3f8e?s=128

Masayuki Imamura

December 09, 2016
Tweet

Transcript

  1. Multi-Cloud Architecture Supporting Machine Learning and Data Analysis ػցֶशͱσʔλ෼ੳΛࢧ͑ΔϚϧνΫϥ΢υΞʔΩςΫνϟͷ঺հ VASILY,Inc.

    @kyuns 2016.12.9 / Minami Aoyama Night #1
  2. ࠓଜխ޾ @kyuns / Ωϡϯ VASILY,Inc औక໾CTO / Co-Founder ೥:BIPP+"1"/ʹೖࣾɺ:BIPP'"4)*0/΍ 9#3"/%ͳͲͷαʔϏεͷ্ཱͪ͛ͷ։ൃΛ୲౰ɻ

    ϦίϝϯσʔγϣϯͷಛڐͳͲΛऔಘޙɺ ೥ʹಠཱɺ7"4*-:Λ૑ۀɻऔక໾$50ʹब೚
  3. ઃཱ೥݄ ॴࡏ஍౦ژ౎඼઒۠੢ޒ൓ాδχΞεϏϧ' ैۀһ໊ ࢿຊۚԯ
 ୅දऔక໾ۚࢁ༟थ
 औక໾ࠓଜխ޾ઍ༿େี 4ZOάϧʔϓ
 ϑΝογϣϯʷςΫϊϩδʔ ΛڧΈͱͨ͠ձࣾ

  4. ೔ຊதͷϑΝογϣϯ ECαΠτͷ঎඼σʔλΛܝࡌ ೔ຊ࠷େڃͷঁͷࢠͷͨΊͷ ϑΝογϣϯΞϓϦʮΞΠίϯʯ ձһ਺ 250ສਓ Google Play ϕετɾΞϓϦ 300Ҏ্ͷ&$αΠτ͔Β

    ສ఺ͷΞΠςϜܝࡌ
  5. ೥࿈ଓϕετΞϓϦड৆ (PPHMF1MBZϕετΠϊϕʔςΟϒΞϓϦେ৆ड৆

  6. ػցֶशͱσʔλ෼ੳΛࢧ͑Δ ϚϧνΫϥ΢υΞʔΩςΫνϟʔͷ঺հ ຊ೔ͷΞδΣϯμ ֤Ϋϥ΢υͷಛੑΛ׆͔ͯ͠
 ༻్ผʹ࠷దͳ΋ͷΛར༻͢Δ

  7. ฐࣾͷσʔλΛ׆༻ͨ͠࠷ۙͷऔΓ૊Έ

  8. 1.σʔλ෼ੳ ΞϓϦ಺ߦಈϩά΍Ϣʔβʔͷσʔλͷ෼ੳ Λ௒͑Δ7JFXͰͦΕͧΕͷࢪࡦͷ,1*Λ֬ೝ

  9. 2.ػցֶश ສਓ͇ສ఺ͷΞΠςϜ͔Β Ϣʔβʔͷ޷ΈΛֶशͯ͠ΞΠςϜΛਪન͢ΔϨίϝϯυΤϯδϯ

  10. 2.ػցֶशʢσΟʔϓϥʔχϯάʣ ը૾͔ΒΞΠςϜݕग़ྨࣅը૾ݕࡧ IUUQUFDIWBTJMZKQFOUSZEFUFDUJPO@BOE@SFUSJFWBM

  11. 2.ػցֶशʢσΟʔϓϥʔχϯάʣ 7"&ͱ("/Λར༻ͨ͠ϑΝογϣϯΞΠςϜݕࡧ *#*4  IUUQUFDIWBTJMZKQFOUSZSFUSJFWBM@VTJOH@WBF@HBO

  12. ϚϧνΫϥ΢υ αʔϏεͷϝΠϯͱͳΔΠϯϑϥ σʔλ෼ੳج൫ͱͯ͠ͷΠϯϑϥ

  13. σʔλ෼ੳ

  14. RDS/Aurora Log Aggregater GoogleDataProc MongoDB GoogleDrive (SpreadSheet) API Server Web

    Server fluentd fluentd fluentd iQONͷϩάσʔλ·ΘΓͷΞʔΩςΫνϟ iOS/Android Puree AWS GCP BigQuery ΦϯϓϨ Tesla K Series / GTX1080
  15. σʔλ෼ੳʹඞཁͳϩάऩू ΞϓϦ "OESPJE͸1VSFFͱJ04͸ಠࣗͷ4%,Λ௨ͯ͠ߦಈϩάΛऔಘ αʔόʔ Πϕϯτϩά͸શͯqVFOUEܦ༝Ͱ#JH2VFSZʹ4USFBNJOH*OTFSU 8FC΍"1*ͷ/HJOYͷϩά΋શͯ4USFBNJOH*OTFSU Ϛελσʔλ 3%4 .POHP%#ʹೖ͍ͬͯΔσʔλ΋͢΂ͯ#JH2VFSZʹίϐʔ IUUQTHJUIVCDPNDPPLQBEQVSFFBOESPJE

  16. (PPHMF͕ఏڙ͢ΔϑϧϚωʔδυ%8)
 ѹ౗తʹ؆୯଎͍͍҆ J20/Ͱ͸ҎԼͷΑ͏ͳσʔλશͯΛอଘ w ΞϓϦ1$εϚϗͷߦಈϩά w Ϛελʔ%#σʔλ GSPN3%4  w

    Ϋϩʔϥʔͷશͯͷϩά w ϦίϝϯυΤϯδϯͷܭࢉ݁Ռ w σΟʔϓϥʔχϯά༻σʔλ BigQuery
  17. ೔ςʔϒϧ͝ͱʹશͯͷϩάΛอଘ IUUQUFDIWBTJMZKQFOUSZCJHRVFSZ@EBUB@QMBUGPSN ྫ:ΞϓϦߦಈϩά

  18. ࠷ۙͷBigQueryͷΞοϓσʔτ w 4UBOEBSE42-ରԠ w */4&3561%"5&%&-&5&΋Մೳʹ w Ұ࣌7JFX഑ྻߏ଄ମαϒΫΤϦͷॻ͚Δ৔ॴ ͕૿͑ͨɺͳͲ༷ʑͳػೳ͕ར༻Մೳʹͳͬͨ w ଞͷػೳมߋͳͲ͸ҎԼʹ·ͱΊͯ͋Γ·͢


    IUUQUFDIWBTJMZKQFOUSZCJHRVFSZTUBOEBSETRM
  19. ؾʹͳΔྉۚ w ετϨʔδ(#
 ετϦʔϛϯάΠϯαʔτ(# w ΫΤϦྔ5#
 
 ஫ҙࣄ߲
 ೥݄͔Β͸ΫΤϦͷ಺༰ʹΑͬͯ஋ஈ͕มΘΔ
 NBYJNVN#JMMJOH5JFSΛઃఆਃ੥͢Δඞཁ͕͋Δ

    ࢀߟJ20/ͷ݄ؒར༻ঢ়گ ೥݄ 
 ෼ੳΫΤϦྔ5JC 5# 
  20. σʔλ෼ੳ͸Tableau #JH2VFSZ΍3%4΋σʔλιʔεͱͯ͠ϦΞϧλΠϜʹѻ͑Δ
 ΠϯλϥΫςΟϒͳ෼ੳ΍μογϡϘʔυͷ࡞੒͕Մೳ ࡉ͔͍σʔλ෼ੳ͸5BCMFBV%FTLUPQ ࣾ಺Ͱͷσʔλڞ༗ʹ͸5BCMFBV4FSWFS

  21. μογϡϘʔυྫ:ΫϩʔϥʔՔಇ཰

  22. ෼ੳྫ: ΰδϥ(※λϒϩʔͷαϯϓϧͰ͢)

  23. ཚཱ͢ΔBIπʔϧ ҰମͲΕΛ࢖͑͹͍͍ͷ͔ʜʁ

  24. BIπʔϧൺֱ ྉۚ #JH2VFSZ ୳ࡧత෼ੳ ߴػೳ ར఺ ܽ఺ 5BCMFBV%FTLUPQ 4FSWFS ༗ྉ

    ˓ ˓ ˓ େ͖ͳσʔλ΋ಡΈ ࠐΊΔ ߴ͍ %BUB4UVEJP ແྉ ˓ ✕ ✕ ͍҆ ࡞ΕΔϘʔυ͸ͭ· Ͱ FYQMPSBUPSZ ༗ྉ ˚ ˓ ˓ LNFBOT47%͕6*Ͱ Ͱ͖Δ όά͕ଟ͍ɺେ͖͍ σʔλಡΈࠐΊͳ͍ SFEBTI ແྉ ˓ ✕ ˚ ͍҆ όά͕ଟ͍ 2VJDL4JHIU ༗ྉ ✕ ✕ ˚ "84ͱͷ਌࿨ੑ #2ະରԠ 4VQFSTFU ແྉ ✕ ✕ ˚ ͍҆ #2ະରԠ 1PXFS#* ༗ྉ ✕ ˓ ˓ .4ͱͷ਌࿨ੑ #2ະରԠ %PNP ༗ྉ ˓ ˓ ˓ ೔ຊͷେखۀͷಋೖ ࣄྫଟ਺ ߴ͍ ࡉ͔͍ൺֱ͸ޙ೔هࣄʹͯ঺հ༧ఆ
  25. ݸਓతBIπʔϧબͼํͷίπ ࣾ಺ʹ#*ʹৄ͍͠ਓ͕͍Δ͓ۚΛଟগ͔͚ͯ΋͍͍ ˠ5BCMFBV%FTLUPQ 5BCMFBV4FSWFS ˞σʔλ෼ੳ͸ࢪࡦΛ൑அ͢ΔͷʹඇৗʹॏཁͳͷͰɺπʔϧࣗମʹόά͕͋Δ ͜ͱࣗମΫϦςΟΧϧ ͱΓ͓͋͑ͣۚΛ͔͚ͣʹ͍͍ײ͡ʹϨϙʔτΛΈ͍ͨ
 ˠ3FEBTIPS(PPHMF%BUB4UVEJP

  26. όονॲཧ "JS'MPX "JSCOC͕ࣾ։ൃͨ͠όονॲཧ࣮ߦϑϨʔϜϫʔΫ ෳ਺ͷδϣϒͷґଘؔ܎ͷղܾ΍ϦτϥΠॲཧͳͲΛ͏·͘΍ͬͯ͘ΕΔ https://github.com/airbnb/airflow -VJHJ 4QPUJGZ͕ࣾ։ൃͨ͠όονॲཧ࣮ߦϑϨʔϜϫʔΫ "JSqPXΑΓ΋γϯϓϧ https://github.com/spotify/luigi

  27. ػցֶश

  28. ػցֶश·ΘΓͷΞʔΩςΫνϟ جຊํ਑ ɾ"84ͰͰ͖Δ͜ͱ͸"84Ͱ΍Δ ɾ#JH2VFSZʹؔ࿈͢Δ෦෼1ZUIPOͷ෦෼͸($1 ɾ(16ͳͲΫϥ΢υͷ஋ஈ͕ߴͯ͘ɺݕূதʹϦιʔεΛৗ࣌େ ྔফඅ͢Δ΋ͷ͸ΦϯϓϨ
 ɾܭࢉࡁ݁Ռ͸3FEJTʹอଘͯ͠"1*͔Βݺͼग़͢ ɾ֤Ϋϥ΢υؒͷ࣮૷͸#JH2VFSZΛத৺ͱͯ͠ૄ݁߹Λҙࣝ͢Δ

  29. ྨࣅը૾APIͷΞʔΩςΫνϟ " 8 4 ( $ 1 Φ ϯ ϓ

    Ϩ ಛఆͷΞΠςϜʹྨࣅ͍ͯ͠ΔΞΠςϜ͕දࣔ͞ΕΔ ಛ௃ྔநग़εςοϓ ྨࣅ౓ܭࢉεςοϓ ΞΠςϜදࣔεςοϓ
  30. ྨࣅը૾APIͷΞʔΩςΫνϟ "1*4FSWFS &$
 "1*3BJMT BigQuery Tesla / GTX1080 / Chainer

    2.ը૾ΛDLɺಛ௃ྔநग़ GCP Batch / Python ֶश༻σʔλऔಘ " 8 4 ( $ 1 Φ ϯ ϓ Ϩ ը૾ͷಛ௃ྔΛอଘ ಛ௃ྔσʔλऔಘ ྨࣅ౓ܭࢉ ྨࣅը૾ܭࢉ݁Ռอଘ &$ #BUDI3VCZ 3FEJT ܭࢉࡁΈσʔλऔಘ Ωϟογϡ࡞੒ ྨࣅը૾σʔλऔಘ ಛఆͷΞΠςϜʹྨࣅ͍ͯ͠ΔΞΠςϜ͕දࣔ͞ΕΔ
  31. ϨίϝϯυΤϯδϯͷΞʔΩςΫνϟ(όονॲཧ൛) &$
 "1*3BJMT BigQuery GCP Batch / Python ߦಈσʔλऔಘ ਪન಺༰ܭࢉ

    ਪન݁Ռอଘ &$ #BUDI3VCZ 3FEJT ਪન݁Ռऔಘ Ωϟογϡ࡞੒ ਪન݁Ռऔಘ " 8 4 ( $ 1 ೔ճɺϢʔβʔͷᅂ޷ʹ͋ͬͨΞΠςϜ͕දࣔ͞ΕΔ
  32. ϨίϝϯυΤϯδϯͷΞʔΩςΫνϟ(ΦϯϥΠϯॲཧ൛) &$
 "1*3BJMT BigQuery GCP Batch / Python " 8

    4 ( $ 1 ߦಈσʔλऔಘ ਪન಺༰ܭࢉ ਪન݁Ռอଘ GCP ߋ৽API / Flask ਪન݁Ռܭࢉˍߋ৽ ߦಈ݁Ռૹ৴ 3&45 ਪન݁Ռऔಘ ਪન݁Ռऔಘ 3&45 Ϣʔβʔ͕ΞΠςϜΛݟΔͨͼʹΦεεϝΞΠςϜ͕มΘͬͯදࣔ͞ΕΔ 3FEJT
  33. ·ͱΊ w σʔλ෼ੳ
 #JH2VFSZ 5BCMFBVͷ૊Έ߹Θ͕ͤͱͯ΋ྑ͍ w ػցֶश
 ಛ௃ྔநग़ͳͲͷॏ͍ॲཧͷաఔ͸ΦϯϓϨͷ(16͕ Φεεϝ
 ͋ͱ͸($1

    "84ͷಛੑΛ׆͔ͯ͠ΞʔΩςΫνϟΛ ߟ͑Δ
 #JH2VFSZΛத৺ʹ͢Δͱઃܭ͠΍ͯ͘͢ӡ༻͠΍͍͢
  34. We’re Hiring! ͋Γ͕ͱ͏͍͟͝·ͨ͠ 7"4*-:Ͱ͸ ϑΝογϣϯʷςΫϊϩδʔʹ ڵຯͷ͋ΔΤϯδχΞΛืू͍ͯ͠·͢ JOGP!WBTJMZKQ