Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Agoop における AWS をフル活用したビッグデータ収集・分析の仕組み / Agoop big data collection and analysis system fully leveraging AWS

Agoop における AWS をフル活用したビッグデータ収集・分析の仕組み / Agoop big data collection and analysis system fully leveraging AWS

Transcript

  1. "84Λϑϧ׆༻ͨ͠ Ϗοάσʔλऩूɾ෼ੳ  "HPPQ$PSQ

  2. Ґஔ৘ใऩू ղੳ ਓͷྲྀΕ

  3. 5FDI1BSU

  4. 4FMGJOUSPEVDUJPO தཬߒ೭ /BLB[BUP)JSPZVLJ ϩάऩूγεςϜͷҡ࣋ɾվળ ΞϓϦͷόοΫΤϯυ։ൃ ΤϯδχΞνʔϜͷ։ൃɾ෼ੳͷͨΊͷ ֤छ࢓૊Έͷ੔උ גࣜձࣾ"HPPQ ٕज़ຊ෦"*։ൃ෦ όοΫΤϯυνʔϜϦʔμʔ

  5. ϩάऩूɾ෼ੳγεςϜͷΞʔΩςΫνϟ "84ΞφϦςΟΫεαʔϏεͷϢʔεέʔε "HFOEB

  6. ϩάऩूɾ෼ੳγεςϜ ΞʔΩςΫνϟ

  7. ೥͔Βӡ༻։࢝ ͢΂ͯࣗલͰ։ൃ ݱࡏ໊ͷόοΫΤϯυνʔϜͰ։ൃ ͢΂ͯ"84্ͰՔಇ ౰ॳ͸&$ɺ4ɺ3%4ͳͲϕʔγοΫͳαʔϏεͰߏங ৽ن"84αʔϏεΛܧଓతɾੵۃతʹಋೖ ϩάྔ͸૿͑ଓ͚Δ ίετޮ཰ͱ҆ఆੑΛ޲্ͤ͞ΔͨΊ γεςϜͷجຊ৘ใ

  8. ϩάσʔλͷ֓ཁ εϚʔτϑΥϯΞϓϦ͔Βऩू த਎͸௨৴ঢ়گͳͲʹؔ͢Δϩά +40/ܗࣜ ೔Ͱऩू͢ΔσʔλϘϦϡʔϜ ݅਺਺ԯ݅ σʔλαΠζ5#௒ γεςϜ͕ऩू͢Δϩά

  9. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB

    1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF
  10. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB

    1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF ऩूɾ஝ੵ
  11. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB

    1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF &5-
  12. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB

    1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF ෼ੳ
  13. ऩूɾ஝ੵ +BWB൛ͷ,JOFTJT1SPEVDFS-JCSBSZͱ,JOFTJT$POTVNFS-JCSBSZΛར༻ ,JOFTJTϓϩσϡʔαʔ͸ड৴ͨ͠+40/Λ΄΅ͦͷ··,JOFTJTʹ౤ೖ ,JOFTJTʹೖΕ͓͚ͯ͹࣌ؒ ࠷௕೔ؒ σʔλϩετͷ৺഑͕ແ͍ͨΊ ,JOFTJTίϯγϡʔϚʔ͸ҎԼͷॲཧΛ࣮ࢪ +40/Λύʔεͯ͠ޙଓͷ෼ੳͰ࢖༻͢Δ৘ใΛ෇Ճ &$಺ͰՔಇ͢ΔqVFOUEʹόοϑΝͯ͠ҰఆִؒͰ4ʹ([JQܗࣜͰग़ྗ <1PJOU>,JOFTJTಋೖͰ҆ఆੑ͕͔ͳΓ޲্ˠσʔλऩूܥʹ͓͢͢Ί

    "VUP4DBMF&$ ,JOFTJT "VUP4DBMF&$ 4
  14. &5- &.3Ϋϥελʔىಈ 4QBSLPO&.3 +FOLJOTPO&$ -BNCEB ࣌ؒʹճ -BNCEBؔ਺Λίʔϧ &.3ΫϥελʔΛىಈ

  15. &5- &.3Ϋϥελʔىಈ <2">ͳͥ$MPVE8BUDI&WFOUTͰ͸ͳ͘+FOLJOTʁ +FOLJOTΛδϣϒεέδϡʔϥʔ୅ΘΓʹ࢖͏จԽ ӡ༻νʔϜ͕+FOLJOTͷऔΓѻ͍ʹ׳Ε͓ͯΓϦΧόϦʔͷ͠΍͢͞Λ༏ઌ <༨ஊ>ෳ਺νʔϜɾෳ਺༻్Ͱ+FOLJOTΛ࢖͏ͱΧΦεԽ͠΍͍͢ ᷖᮣʹ04ύοέʔδ΍ϓϥάΠϯΛΞοϓσʔτ͢Δͱ஌Βͳ͍δϣϒ͕ࣦഊ ͍ͭ΋Կ͔ͷδϣϒ͕ಈ͍͍ͯͯϓϥάΠϯͷΠϯετʔϧ͕Ͱ͖ͳ͍FUDʜ +FOLJOT͔Βͷ୤٫Λ໛ࡧதˠ$MPVE8BUDI&WFOUT "84#BUDIΛݕ౼

    4QBSLPO&.3 +FOLJOTPO&$ -BNCEB ࣌ؒʹճ -BNCEBؔ਺Λίʔϧ &.3ΫϥελʔΛىಈ
  16. &5- &.3Ϋϥελʔىಈ <2">ͳͥ-BNCEBؔ਺ʁ ༷ʑͳνʔϜͰ&.3ΫϥελʔΛؾܰʹར༻ͯ͠΋Β͏ͨΊ &5-ॲཧҎ֎ʹ΋༷ʑͳνʔϜɾ༻్Ͱ&.3Λར༻͍ͯ͠Δ &.3Ϋϥελʔͷىಈ͸ύϥϝʔλʔ͕ଟ͘Ұ͔Βىಈ͢Δͷ͸େม Ϋϥελʔىಈͷڞ௨తͳॲཧΛ1ZUIPO CPUP Ͱ࣮૷ ֤νʔϜ͔Βͷཁ๬ΛݩʹΫϥελʔͷεϖοΫ౳ΛઃఆϑΝΠϧʹهड़͠1ZUIPO

    ϞδϡʔϧʹؚΊͯ-BNCEBؔ਺ͱͯ͠Ξοϓϩʔυ ֤νʔϜ͸ઃఆϑΝΠϧΛࢦఆ͠-BNCEBؔ਺Λݺͼग़͢͜ͱͰΫϥελʔىಈ 4QBSLPO&.3 +FOLJOTPO&$ -BNCEB ࣌ؒʹճ -BNCEBؔ਺Λίʔϧ &.3ΫϥελʔΛىಈ
  17. &5- 3FETIJGUɾ)JWFϩʔυ ݱࡏ͸.Z42-3%4Λ)JWFϝλετΞ ςʔϒϧఆٛ౳ͷ؅ཧઌ ͱͯ͠ར༻ (MVFσʔλΧλϩά΁ͷҠߦɾ੾ସΛ࣮ࢪத 3%4͔Β(MVF΁ͷҠߦ͸(JU)VCͷBXTMBCTͷπʔϧΛར༻ IUUQTHJUIVCDPNBXTMBCTBXTHMVFTBNQMFTUSFFNBTUFS VUJMJUJFT)JWF@NFUBTUPSF@NJHSBUJPO ॳ෺ͷέʔε͕͍͔ͭ͋͘Γগ͠ϋϚ͕ͬͨ"84αϙʔτͷ͓ӄͰղܾʂ

    4QBSL PO&.3 4 ([JQ 4 1BSRVFU 3FETIJGU 3%4 )JWF.FUBTUPSF ऩूɾ஝ੵ 4
  18. &5- 3FETIJGUɾ)JWFϩʔυ <1PJOU>)JWFϝλετΞ͸3%4ΑΓ΋(MVF%BUB$BUBMPH͕͓͢͢Ί 3%4͕410' ୯Ұো֐఺ ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋Δ 3%4͸ಉ࣌઀ଓ਺ͷ੍ݶ͕͋ΔͨΊ&.3Ϋϥελʔͷಉ࣌ىಈ਺ʹ੍ݶ͕ੜ͡Δ ࠷େಉ࣌઀ଓ਺͸ϝϞϦґଘͷઃఆ ࠷େಉ࣌઀ଓ਺Λ্͛Δʹ͸ΑΓϝϞϦ͕ଟ͍ΠϯελϯελΠϓ΁ͷมߋ͕ඞཁ ϝλσʔλ͔͠؅ཧ͠ͳ͍ͷʹϦονͳεϖοΫ͸໪ମແ͍

    (MVF%BUB$BUBMPH͸ϑϧϚωʔδυ͔ͭಉ࣌઀ଓ਺ͷ੍ݶͳ͠ 4QBSL PO&.3 4 ([JQ 4 1BSRVFU 3FETIJGU 3%4 )JWF.FUBTUPSF ऩूɾ஝ੵ 4
  19. &5- 3FETIJGUɾ)JWFϩʔυ 4QBSLPO&.3Ͱ͸࣌ؒ෼ͷϩάΛೖྗͱͯ͠ҎԼεςοϓΛ࣮ߦ 4%JTU$QͰ໿ ݸ ,#ڃ ͷϑΝΠϧΛ໿ݸ .#d(#ڃ ͷϑΝΠϧʹू໿ <1PJOU>4QBSL

    )BEPPQ ͸େྔͷখ͞ͳϑΝΠϧΛѻ͏ͷ͸ෆಘҙ 4QBSLͰ([JQQFE+40/Λग़ྗ)JWFςʔϒϧʹ*/4&3507&383*5& εςοϓऴྃޙɺ&.3Ϋϥελʔ͸ࣗಈ࡟আ ࣌ؒ෼ͷॲཧΛಠཱͨ͠ΫϥελʔͰ࣮ߦ͢ΔͷͰϦΧόϦʔͳͲ͕΍Γ΍͍͢ 4্ͷ([JQϑΝΠϧΛ3FETIJGUͷ$01:ίϚϯυͰϩʔυ 4QBSL PO&.3 4 ([JQ 4 1BSRVFU 3FETIJGU 3%4 )JWF.FUBTUPSF ऩूɾ஝ੵ 4
  20. σʔλऩूܥʹ͸,JOFTJT4USFBN͕͓͢͢Ί )JWFϝλετΞ͸(MVF%BUB$BUBMPH͕͓͢͢Ί 3%4͔Β(MVF΁ͷҠߦʹ͸(JU)VCͷBXTMBCTͷπʔϧ͕࢖͑Δ IUUQTHJUIVCDPNBXTMBCTBXTHMVFTBNQMFTUSFFNBTUFSVUJMJUJFT )JWF@NFUBTUPSF@NJHSBUJPO 4QBSL )BEPPQ ͸େྔͷখ͍͞ϑΝΠϧΛѻ͏ͷ͸ෆಘҙͳͨΊɺ 4%JTU$QͳͲΛ༻͍ͯ͋Δఔ౓ͷେ͖͞ʹ·ͱΊΔͷ͕͓͢͢Ί &.3ϕετϓϥΫςΟεͰ͸([JQѹॖͷ৔߹ɺʙ(#ϑΝΠϧΛਪ঑

    IUUQTNFEJBBNB[POXFCTFSWJDFTDPNKQXQ "84@"NB[PO@&.3@#FTU@1SBDUJDFTQEG લ൒ͷ·ͱΊ
  21. "84ΞφϦςΟΫεαʔϏε Ϣʔεέʔε

  22. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB

    1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF ෼ੳ
  23. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB

    1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF 3FETIJGU +FOLJOT ఆظఆܕ෼ੳ
  24. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB

    1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF "UIFOB1SFTUPPO&.3 3FEBTI ΞυϗοΫ෼ੳ
  25. 3FETIJGU ఆظఆܕ෼ੳ͕த৺ +FOLJOTΛδϣϒεέδϡʔϥʔͱͯ͠ར༻ σʔλαΠζͱϫʔΫϩʔυʹ߹ͬͨεϖοΫͷ3*Λߪೖ 3*ͷظݶ੾Ε͕͍ۙͨΊ)JWFPO&.3΁ͷҠߦΛਪਐத ΞφϦςΟΫεαʔϏεͷ࢖͍෼͚

  26. "UIFOB 1SFTUPPO&.3 ΞυϗοΫ෼ੳ͕த৺ 3FEBTI 044ͷμογϡϘʔυπʔϧ Λ(6*ͱͯ͠ར༻ ೥ॳ಄͔Β3FEBTI "UIFOBಋೖ ݩʑ͸3FETIJGUͰΞυϗοΫ෼ੳ΋ߦͳ͍ͬͯͨ σʔλྔ૿Ճɺूܭύλʔϯ૿Ճ

    ఆظఆܕ෼ੳͱΞυϗοΫ෼ੳͰͷϦιʔεͷୣ͍߹͍ ΞφϦςΟΫεαʔϏεͷ࢖͍෼͚
  27. ͳͥ"UIFOBͱ1SFTUPPO&.3ͷ྆ํΛར༻ʁ "UIFOBͷར༻ྉ͕ۚ૝ఆΑΓߴ͘ͳͬͯ͠·ͬͨ ྉۚ  ݄ εΩϟϯαΠζ1# ݄ ࣍ળͷࡦͱͯ͠1SFTUPPO&.3Λಋೖ ΞφϦςΟΫεαʔϏεͷ࢖͍෼͚

  28. ΞυϗοΫ෼ੳͷ೥ද ೥ 3FETIJGU ར༻։࢝ ೥ ݄ w 3FEBTIར༻։࢝ w σʔλιʔεʹ

    3FETIJGUͱ "UIFOBΛ௥Ճ ݄ 3FEBTIͷ σʔλιʔε͔Β 3FETIJGUΛ࡟আ ݄ ✖3FETIJGUෛՙ૿େ ✔ࣾ಺ʹ޿͘ීٴ ✖"UIFOBͷ ར༻ྉ͕ۚ   ݄ 1# εΩϟϯ w 1SFTUPPO&.3 ར༻։࢝ w "UIFOBͷΫΤϦ Λ1SFTUPʹҠߦ ✖ੑೳ໘Ͱෆຬ ✔ར༻ྉ͕ۚ   ݄Ͱ҆ఆ ݱࡏ ΞυϗοΫ෼ੳͷ සग़ͷूܭॲཧΛ )JWFPO&.3Ͱ όονॲཧԽ ݱࡏਐߦத ೥ ݄
  29. ϋʔυ΢ΣΞϦιʔεΛҙࣝͤͣʹ5#ΦʔμʔͷεΩϟϯΛݱ࣮తͳ࣌ؒͰॲཧͰ͖Δ ͋Δఔ౓ʮແ஡ʯͳΫΤϦ΋"UIFOBͳΒݱ࣮తͳ࣌ؒͰॲཧͰ͖ͯ͠·͏ "UIFOBͷ࢖͍উखͷྑ͞ ԯ݅ͷϩάΛ࡞੒೔ผʹूܭ "UIFOB෼ඵͰ׬ྃ εΩϟϯαΠζ(# 1SFTUPPO&.3෼Ͱ׬ྃ NYMBSHF୆ .BTUFS $PSF

    5BTL ෼ඵ ෼ "UIFOB 1SFTUPPO&.3
  30. "UIFOBͷ࢖͍উखͷྑ͞ ϋʔυ΢ΣΞϦιʔεͷ੍໿ "UIFOB σϑΥϧτͷಉ࣮࣌ߦΫΤϦ਺ͷ্ݶ͸ ಉ࣮࣌ߦͨ͠ΫΤϦؒͰϦιʔεͷୣ͍߹͍͸ൃੜ͠ͳ͍ 1SFTUPPO&.3 ΫϥελʔͷϝϞϦͳͲͷϦιʔεΛෳ਺ΫΤϦͰγΣΞ͢Δ ԼهͷϝϞϦઃఆΛ௒͑ΔΫΤϦ͸࣮ߦͰ͖ͳ͍ ઃఆ߲໨ આ໌

    RVFSZNBYNFNPSZ ΫΤϦͰ࢖༻͢ΔϝϞϦͷ࠷େ஋ RVFSZNBYNFNPSZQFSOPEF ΫΤϦϊʔυͰ࢖༻͢ΔϝϞϦͷ࠷େ஋
  31. "UIFOBͷར༻ྉۚߴಅͷݪҼ 3FEBTIͷΫΤϦεέδϡʔϧػೳ 3FEBTIʹอଘͨ͠ΫΤϦΛࢦఆͨ͠ස౓Ͱ࣮ߦͰ͖Δ "UIFOBͷ࢖͍উखͷྑ͞ ͋Δఔ౓ʮແ஡ʯͳΫΤϦ΋ݱ࣮తͳ࣌ؒͰॲཧͰ͖Δ ਺5#ΛεΩϟϯ͢ΔΫΤϦ܈͕සൟʹ࣮ߦ ߹ܭεΩϟϯαΠζ͕1#ΛΦʔόʔ

  32. ݱࡏ͸1SFTUPPO&.3ΛϝΠϯͰར༻ ୆ن໛Ͱ΋"UIFOBʹશવఢΘͳ͍ʜ εΩϟϯαΠζΛখ͘͢͞ΔͨΊɺසग़ͷूܭॲཧΛ )JWFPO&.3ͰόονॲཧԽ͢Δऔ૊Λਪਐத ্هऔ૊͕׬ྃͨ͠Β"UIFOBͷํ͕҆͘ͳΔՄೳੑ͕ ͋ΔͨΊɺ"UIFOBΛϝΠϯʹ͢Δ͜ͱ΋ݕ౼ ΞυϗοΫ෼ੳͷݱࡏͱࠓޙ

  33. ϏοάσʔλͷΞυϗοΫ෼ੳͷจ຺ʹ͓͍ͯ "UIFOB͸࠷ڧͷબ୒ࢶͷҰͭ ಛʹ੍ݶΛઃ͚ͣʹ"UIFOBΛಋೖ͢Δͱ૝ఆΑΓར༻ྉ͕ۚߴ͘ ͳΔέʔε͕͋Δ 3FEBTIͱͷ૊߹ͤͰ"UIFOBීٴͷϋʔυϧ͸άϯͱԼ͕Δ͕ɺ ΫΤϦεέδϡʔϧػೳͳͲ͸஫ҙ͕ඞཁ "84ར༻ྉۚΛීஈ͋·Γҙࣝ͠ͳ͍Ϣʔβʔ͕"UIFOBΛར༻ ͢Δ৔߹ɺίετҙࣝͳͲΛఆணͤ͞Δ׆ಈ͕ඞཁ ޙ൒ͷ·ͱΊɾڭ܇

  34. ޙ൒ͷ·ͱΊɾڭ܇ ίετҙࣝ ΫΤϦ࣮ߦʹΑΓԿ(#εΩϟϯ͢Δͷ͔ɺͦͷ݁Ռྉۚ͸͍͘Βֻ͔Δ͔Λ Ϣʔβʔ͕ҙࣝ͢Δ͜ͱ͕ඇৗʹॏཁ "84؅ཧऀ޲͚ Ϣʔβʔ͕ར༻ྉۚΛ֬ೝͰ͖Δ࢓૊ΈΛ༻ҙ͢Δ %3:ݪଇ %PO`U3FQFBU:PVSTFMG  εΩϟϯαΠζ͕େ͖͍ΫΤϦΛԿճ΋࣮ߦ͠ͳ͍ʗͤ͞ͳ͍

    සग़ͷΫΤϦ͸ճ͚࣮ͩߦͯ݁͠ՌΛอଘ͢Δ ͓࡞๏ͷపఈ εΩϟϯαΠζ௿ݮͷͨΊύʔςΟγϣϯΛඞͣࢦఆ͢Δ ඞཁ࠷௿ݶͷΧϥϜΛ4&-&$5͢Δ ҆қʹ4&-&$5 ͠ͳ͍ FUD "UIFOBΛίετޮ཰ྑ͘׆༻͢ΔͨΊͷϚΠϯυηοτ
  35. &0'