Agoop における AWS をフル活用したビッグデータ収集・分析の仕組み / Agoop big data collection and analysis system fully leveraging AWS

Transcript

1. "84Λϑϧ׆༻ͨ͠
   Ϗοάσʔλऩूɾ෼ੳ 
   "HPPQ
   $PSQ
2. Ґஔ৘ใऩू ղੳ ਓͷྲྀΕ
3. 5FDI
   1BSU
4. 4FMGJOUSPEVDUJPO தཬ
   ߒ೭ /BLB[BUP
   )JSPZVLJ ϩάऩूγεςϜͷҡ࣋ɾվળ
   ΞϓϦͷόοΫΤϯυ։ൃ
   ΤϯδχΞνʔϜͷ։ൃɾ෼ੳͷͨΊͷ
   ֤छ࢓૊Έͷ੔උ גࣜձࣾ"HPPQ
   ٕज़ຊ෦
   "*։ൃ෦
   όοΫΤϯυνʔϜϦʔμʔ
5. ϩάऩूɾ෼ੳγεςϜͷΞʔΩςΫνϟ
   "84ΞφϦςΟΫεαʔϏεͷϢʔεέʔε "HFOEB
6. ϩάऩूɾ෼ੳγεςϜ
   ΞʔΩςΫνϟ
7. ೥͔Βӡ༻։࢝
   ͢΂ͯࣗલͰ։ൃ
   ݱࡏ໊ͷόοΫΤϯυνʔϜͰ։ൃ
   ͢΂ͯ"84্ͰՔಇ
   ౰ॳ͸&$ɺ4ɺ3%4ͳͲϕʔγοΫͳαʔϏεͰߏங
   ৽ن"84αʔϏεΛܧଓతɾੵۃతʹಋೖ
   ϩάྔ͸૿͑ଓ͚Δ
   ίετޮ཰ͱ҆ఆੑΛ޲্ͤ͞ΔͨΊ γεςϜͷجຊ৘ใ
8. ϩάσʔλͷ֓ཁ
   εϚʔτϑΥϯΞϓϦ͔Βऩू
   த਎͸௨৴ঢ়گͳͲʹؔ͢Δϩά
   +40/ܗࣜ
   ೔Ͱऩू͢ΔσʔλϘϦϡʔϜ
   ݅਺
   ਺ԯ݅
   σʔλαΠζ
   5#௒ γεςϜ͕ऩू͢Δϩά
9. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
   &$ ,JOFTJT "VUP4DBMJOH
   &$ 4 4 3FETIJGU "UIFOB 1SFTUP
   PO
   &.3 4 4QBSL
   PO
   &.3 (MVF +FOLJOT
   PO
   &$ -BNCEB %FWFMPQFST
   %BUB
   "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
   )JWF
   .FUBTUPSF
10. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF +FOLJOT
    PO
    &$ -BNCEB %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF ऩूɾ஝ੵ
11. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF +FOLJOT
    PO
    &$ -BNCEB %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF &5-
12. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF +FOLJOT
    PO
    &$ -BNCEB %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF ෼ੳ
13. ऩूɾ஝ੵ +BWB൛ͷ,JOFTJT
    1SPEVDFS
    -JCSBSZͱ,JOFTJT
    $POTVNFS
    -JCSBSZΛར༻
    ,JOFTJTϓϩσϡʔαʔ͸ड৴ͨ͠+40/Λ΄΅ͦͷ··,JOFTJTʹ౤ೖ
    ,JOFTJTʹೖΕ͓͚ͯ͹࣌ؒ
    ࠷௕೔ؒ
    σʔλϩετͷ৺഑͕ແ͍ͨΊ
    ,JOFTJTίϯγϡʔϚʔ͸ҎԼͷॲཧΛ࣮ࢪ
    +40/Λύʔεͯ͠ޙଓͷ෼ੳͰ࢖༻͢Δ৘ใΛ෇Ճ
    &$಺ͰՔಇ͢ΔqVFOUEʹόοϑΝͯ͠ҰఆִؒͰ4ʹ([JQܗࣜͰग़ྗ
    <1PJOU>
    ,JOFTJTಋೖͰ҆ఆੑ͕͔ͳΓ޲্ˠσʔλऩूܥʹ͓͢͢Ί "VUP4DBMF
    &$ ,JOFTJT "VUP4DBMF
    &$ 4
14. &5-
    &.3Ϋϥελʔىಈ 4QBSL
    PO
    &.3 +FOLJOT
    PO
    &$ -BNCEB ࣌ؒʹճ
    -BNCEBؔ਺Λίʔϧ &.3ΫϥελʔΛىಈ
15. &5-
    &.3Ϋϥελʔىಈ <2">
    ͳͥ$MPVE8BUDI
    &WFOUTͰ͸ͳ͘+FOLJOTʁ
    +FOLJOTΛδϣϒεέδϡʔϥʔ୅ΘΓʹ࢖͏จԽ
    ӡ༻νʔϜ͕+FOLJOTͷऔΓѻ͍ʹ׳Ε͓ͯΓϦΧόϦʔͷ͠΍͢͞Λ༏ઌ
    <༨ஊ>
    ෳ਺νʔϜɾෳ਺༻్Ͱ+FOLJOTΛ࢖͏ͱΧΦεԽ͠΍͍͢
    ᷖᮣʹ04ύοέʔδ΍ϓϥάΠϯΛΞοϓσʔτ͢Δͱ஌Βͳ͍δϣϒ͕ࣦഊ
    ͍ͭ΋Կ͔ͷδϣϒ͕ಈ͍͍ͯͯϓϥάΠϯͷΠϯετʔϧ͕Ͱ͖ͳ͍
    FUDʜ
    +FOLJOT͔Βͷ୤٫Λ໛ࡧதˠ$MPVE8BUDI
    &WFOUT
    
    "84
    #BUDIΛݕ౼ 4QBSL
    PO
    &.3 +FOLJOT
    PO
    &$ -BNCEB ࣌ؒʹճ
    -BNCEBؔ਺Λίʔϧ &.3ΫϥελʔΛىಈ
16. &5-
    &.3Ϋϥελʔىಈ <2">
    ͳͥ-BNCEBؔ਺ʁ
    ༷ʑͳνʔϜͰ&.3ΫϥελʔΛؾܰʹར༻ͯ͠΋Β͏ͨΊ
    &5-ॲཧҎ֎ʹ΋༷ʑͳνʔϜɾ༻్Ͱ&.3Λར༻͍ͯ͠Δ
    &.3Ϋϥελʔͷىಈ͸ύϥϝʔλʔ͕ଟ͘Ұ͔Βىಈ͢Δͷ͸େม
    Ϋϥελʔىಈͷڞ௨తͳॲཧΛ1ZUIPO
    CPUP
    Ͱ࣮૷
    ֤νʔϜ͔Βͷཁ๬ΛݩʹΫϥελʔͷεϖοΫ౳ΛઃఆϑΝΠϧʹهड़͠1ZUIPO ϞδϡʔϧʹؚΊͯ-BNCEBؔ਺ͱͯ͠Ξοϓϩʔυ
    ֤νʔϜ͸ઃఆϑΝΠϧΛࢦఆ͠-BNCEBؔ਺Λݺͼग़͢͜ͱͰΫϥελʔىಈ 4QBSL
    PO
    &.3 +FOLJOT
    PO
    &$ -BNCEB ࣌ؒʹճ
    -BNCEBؔ਺Λίʔϧ &.3ΫϥελʔΛىಈ
17. &5-
    3FETIJGUɾ)JWFϩʔυ ݱࡏ͸.Z42-
    3%4Λ)JWFϝλετΞ
    ςʔϒϧఆٛ౳ͷ؅ཧઌ
    ͱͯ͠ར༻
    (MVFσʔλΧλϩά΁ͷҠߦɾ੾ସΛ࣮ࢪத
    3%4͔Β(MVF΁ͷҠߦ͸(JU)VCͷBXTMBCTͷπʔϧΛར༻
    IUUQTHJUIVCDPNBXTMBCTBXTHMVFTBNQMFTUSFFNBTUFS VUJMJUJFT)JWF@NFUBTUPSF@NJHSBUJPO
    ॳ෺ͷέʔε͕͍͔ͭ͋͘Γগ͠ϋϚ͕ͬͨ"84αϙʔτͷ͓ӄͰղܾʂ 4QBSL
    PO
    &.3 4
    ([JQ 4
    1BSRVFU 3FETIJGU 3%4
    )JWF
    .FUBTUPSF ऩूɾ஝ੵ 4
18. &5-
    3FETIJGUɾ)JWFϩʔυ <1PJOU>
    )JWFϝλετΞ͸3%4ΑΓ΋(MVF
    %BUB
    $BUBMPH͕͓͢͢Ί
    3%4͕410'
    ୯Ұো֐఺
    ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋Δ
    3%4͸ಉ࣌઀ଓ਺ͷ੍ݶ͕͋ΔͨΊ&.3Ϋϥελʔͷಉ࣌ىಈ਺ʹ੍ݶ͕ੜ͡Δ
    ࠷େಉ࣌઀ଓ਺͸ϝϞϦґଘͷઃఆ
    ࠷େಉ࣌઀ଓ਺Λ্͛Δʹ͸ΑΓϝϞϦ͕ଟ͍ΠϯελϯελΠϓ΁ͷมߋ͕ඞཁ
    ϝλσʔλ͔͠؅ཧ͠ͳ͍ͷʹϦονͳεϖοΫ͸໪ମແ͍
    (MVF
    %BUB
    $BUBMPH͸ϑϧϚωʔδυ͔ͭಉ࣌઀ଓ਺ͷ੍ݶͳ͠ 4QBSL
    PO
    &.3 4
    ([JQ 4
    1BSRVFU 3FETIJGU 3%4
    )JWF
    .FUBTUPSF ऩूɾ஝ੵ 4
19. &5-
    3FETIJGUɾ)JWFϩʔυ 4QBSL
    PO
    &.3Ͱ͸࣌ؒ෼ͷϩάΛೖྗͱͯ͠ҎԼεςοϓΛ࣮ߦ
    4%JTU$QͰ໿ ݸ
    ,#ڃ
    ͷϑΝΠϧΛ໿ݸ
    .#d(#ڃ
    ͷϑΝΠϧʹू໿
    <1PJOU>
    4QBSL
    )BEPPQ
    ͸େྔͷখ͞ͳϑΝΠϧΛѻ͏ͷ͸ෆಘҙ
    4QBSLͰ([JQQFE
    +40/Λग़ྗ
    
    )JWFςʔϒϧʹ*/4&35
    07&383*5&
    εςοϓऴྃޙɺ&.3Ϋϥελʔ͸ࣗಈ࡟আ
    ࣌ؒ෼ͷॲཧΛಠཱͨ͠ΫϥελʔͰ࣮ߦ͢ΔͷͰϦΧόϦʔͳͲ͕΍Γ΍͍͢
    4্ͷ([JQϑΝΠϧΛ3FETIJGUͷ$01:ίϚϯυͰϩʔυ 4QBSL
    PO
    &.3 4
    ([JQ 4
    1BSRVFU 3FETIJGU 3%4
    )JWF
    .FUBTUPSF ऩूɾ஝ੵ 4
20. σʔλऩूܥʹ͸,JOFTJT
    4USFBN͕͓͢͢Ί
    )JWFϝλετΞ͸(MVF
    %BUB
    $BUBMPH͕͓͢͢Ί
    3%4͔Β(MVF΁ͷҠߦʹ͸(JU)VCͷBXTMBCTͷπʔϧ͕࢖͑Δ
    IUUQTHJUIVCDPNBXTMBCTBXTHMVFTBNQMFTUSFFNBTUFSVUJMJUJFT )JWF@NFUBTUPSF@NJHSBUJPO
    4QBSL
    )BEPPQ
    ͸େྔͷখ͍͞ϑΝΠϧΛѻ͏ͷ͸ෆಘҙͳͨΊɺ 4%JTU$QͳͲΛ༻͍ͯ͋Δఔ౓ͷେ͖͞ʹ·ͱΊΔͷ͕͓͢͢Ί
    &.3ϕετϓϥΫςΟεͰ͸([JQѹॖͷ৔߹ɺʙ(#ϑΝΠϧΛਪ঑
    IUUQTNFEJBBNB[POXFCTFSWJDFTDPNKQXQ "84@"NB[PO@&.3@#FTU@1SBDUJDFTQEG લ൒ͷ·ͱΊ
21. "84ΞφϦςΟΫεαʔϏε
    Ϣʔεέʔε
22. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF +FOLJOT
    PO
    &$ -BNCEB %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF ෼ੳ
23. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF +FOLJOT
    PO
    &$ -BNCEB %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF 3FETIJGU
    
    +FOLJOT
    
    ఆظఆܕ෼ੳ
24. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF +FOLJOT
    PO
    &$ -BNCEB %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF "UIFOB
    
    1SFTUP
    PO
    &.3
    
    3FEBTI
    
    ΞυϗοΫ෼ੳ
25. 3FETIJGU
    ఆظఆܕ෼ੳ͕த৺
    +FOLJOTΛδϣϒεέδϡʔϥʔͱͯ͠ར༻
    σʔλαΠζͱϫʔΫϩʔυʹ߹ͬͨεϖοΫͷ3*Λߪೖ
    3*ͷظݶ੾Ε͕͍ۙͨΊ)JWF
    PO
    &.3΁ͷҠߦΛਪਐத ΞφϦςΟΫεαʔϏεͷ࢖͍෼͚
26. "UIFOB
    1SFTUP
    PO
    &.3
    ΞυϗοΫ෼ੳ͕த৺
    3FEBTI
    044ͷμογϡϘʔυπʔϧ
    Λ(6*ͱͯ͠ར༻
    ೥ॳ಄͔Β3FEBTI
    
    "UIFOBಋೖ
    ݩʑ͸3FETIJGUͰΞυϗοΫ෼ੳ΋ߦͳ͍ͬͯͨ
    σʔλྔ૿Ճɺूܭύλʔϯ૿Ճ
    ఆظఆܕ෼ੳͱΞυϗοΫ෼ੳͰͷϦιʔεͷୣ͍߹͍ ΞφϦςΟΫεαʔϏεͷ࢖͍෼͚
27. ͳͥ"UIFOBͱ1SFTUP
    PO
    &.3ͷ྆ํΛར༻ʁ
    "UIFOBͷར༻ྉ͕ۚ૝ఆΑΓߴ͘ͳͬͯ͠·ͬͨ
    ྉۚ
      ݄
    εΩϟϯαΠζ
    1# ݄
    ࣍ળͷࡦͱͯ͠1SFTUP
    PO
    &.3Λಋೖ ΞφϦςΟΫεαʔϏεͷ࢖͍෼͚
28. ΞυϗοΫ෼ੳͷ೥ද ೥ 3FETIJGU
    ར༻։࢝ ೥ ݄ w 3FEBTIར༻։࢝
    w σʔλιʔεʹ 3FETIJGUͱ "UIFOBΛ௥Ճ ݄ 3FEBTIͷ
    σʔλιʔε͔Β
    3FETIJGUΛ࡟আ ݄ ✖
    3FETIJGUෛՙ૿େ ✔
    ࣾ಺ʹ޿͘ීٴ ✖
    "UIFOBͷ
    ར༻ྉ͕ۚ   ݄
    1# εΩϟϯ w 1SFTUP
    PO
    &.3 ར༻։࢝
    w "UIFOBͷΫΤϦ Λ1SFTUPʹҠߦ ✖
    ੑೳ໘Ͱෆຬ ✔
    ར༻ྉ͕ۚ   ݄Ͱ҆ఆ ݱࡏ ΞυϗοΫ෼ੳͷ
    සग़ͷूܭॲཧΛ
    )JWF
    PO
    &.3Ͱ
    όονॲཧԽ
    ݱࡏਐߦத ೥ ݄
29. ϋʔυ΢ΣΞϦιʔεΛҙࣝͤͣʹ5#ΦʔμʔͷεΩϟϯΛݱ࣮తͳ࣌ؒͰॲཧͰ͖Δ
    
    ͋Δఔ౓ʮແ஡ʯͳΫΤϦ΋"UIFOBͳΒݱ࣮తͳ࣌ؒͰॲཧͰ͖ͯ͠·͏ "UIFOBͷ࢖͍উखͷྑ͞ ԯ݅ͷϩάΛ࡞੒೔ผʹूܭ
    "UIFOB
    ෼ඵͰ׬ྃ
    εΩϟϯαΠζ
    (#
    1SFTUP
    PO
    &.3
    ෼Ͱ׬ྃ
    NYMBSHF
    ୆
    .BTUFS
    $PSF
    5BTL ෼ඵ ෼ "UIFOB 1SFTUP
    PO
    &.3
30. "UIFOBͷ࢖͍উखͷྑ͞ ϋʔυ΢ΣΞϦιʔεͷ੍໿
    "UIFOB
    σϑΥϧτͷಉ࣮࣌ߦΫΤϦ਺ͷ্ݶ͸
    ಉ࣮࣌ߦͨ͠ΫΤϦؒͰϦιʔεͷୣ͍߹͍͸ൃੜ͠ͳ͍
    1SFTUP
    PO
    &.3
    ΫϥελʔͷϝϞϦͳͲͷϦιʔεΛෳ਺ΫΤϦͰγΣΞ͢Δ
    ԼهͷϝϞϦઃఆΛ௒͑ΔΫΤϦ͸࣮ߦͰ͖ͳ͍ ઃఆ߲໨ આ໌ RVFSZNBYNFNPSZ ΫΤϦͰ࢖༻͢ΔϝϞϦͷ࠷େ஋ RVFSZNBYNFNPSZQFSOPEF ΫΤϦ
    
    ϊʔυͰ࢖༻͢ΔϝϞϦͷ࠷େ஋
31. "UIFOBͷར༻ྉۚߴಅͷݪҼ 3FEBTIͷΫΤϦεέδϡʔϧػೳ
    3FEBTIʹอଘͨ͠ΫΤϦΛࢦఆͨ͠ස౓Ͱ࣮ߦͰ͖Δ "UIFOBͷ࢖͍উखͷྑ͞
    ͋Δఔ౓ʮແ஡ʯͳΫΤϦ΋ݱ࣮తͳ࣌ؒͰॲཧͰ͖Δ ਺5#ΛεΩϟϯ͢ΔΫΤϦ܈͕සൟʹ࣮ߦ
    ߹ܭεΩϟϯαΠζ͕1#ΛΦʔόʔ
32. ݱࡏ͸1SFTUP
    PO
    &.3ΛϝΠϯͰར༻
    ୆ن໛Ͱ΋"UIFOBʹશવఢΘͳ͍ʜ
    εΩϟϯαΠζΛখ͘͢͞ΔͨΊɺසग़ͷूܭॲཧΛ )JWF
    PO
    &.3ͰόονॲཧԽ͢Δऔ૊Λਪਐத
    ্هऔ૊͕׬ྃͨ͠Β"UIFOBͷํ͕҆͘ͳΔՄೳੑ͕ ͋ΔͨΊɺ"UIFOBΛϝΠϯʹ͢Δ͜ͱ΋ݕ౼ ΞυϗοΫ෼ੳͷݱࡏͱࠓޙ
33. ϏοάσʔλͷΞυϗοΫ෼ੳͷจ຺ʹ͓͍ͯ
    "UIFOB͸࠷ڧͷબ୒ࢶͷҰͭ
    ಛʹ੍ݶΛઃ͚ͣʹ"UIFOBΛಋೖ͢Δͱ૝ఆΑΓར༻ྉ͕ۚߴ͘ ͳΔέʔε͕͋Δ
    3FEBTIͱͷ૊߹ͤͰ"UIFOBීٴͷϋʔυϧ͸άϯͱԼ͕Δ͕ɺ ΫΤϦεέδϡʔϧػೳͳͲ͸஫ҙ͕ඞཁ
    "84ར༻ྉۚΛීஈ͋·Γҙࣝ͠ͳ͍Ϣʔβʔ͕"UIFOBΛར༻ ͢Δ৔߹ɺίετҙࣝͳͲΛఆணͤ͞Δ׆ಈ͕ඞཁ ޙ൒ͷ·ͱΊɾڭ܇
34. ޙ൒ͷ·ͱΊɾڭ܇ ίετҙࣝ
    ΫΤϦ࣮ߦʹΑΓԿ(#εΩϟϯ͢Δͷ͔ɺͦͷ݁Ռྉۚ͸͍͘Βֻ͔Δ͔Λ Ϣʔβʔ͕ҙࣝ͢Δ͜ͱ͕ඇৗʹॏཁ
    "84؅ཧऀ޲͚
    Ϣʔβʔ͕ར༻ྉۚΛ֬ೝͰ͖Δ࢓૊ΈΛ༻ҙ͢Δ
    %3:ݪଇ
    %PO`U
    3FQFBU
    :PVSTFMG
    εΩϟϯαΠζ͕େ͖͍ΫΤϦΛԿճ΋࣮ߦ͠ͳ͍ʗͤ͞ͳ͍
    සग़ͷΫΤϦ͸ճ͚࣮ͩߦͯ݁͠ՌΛอଘ͢Δ
    ͓࡞๏ͷపఈ
    εΩϟϯαΠζ௿ݮͷͨΊύʔςΟγϣϯΛඞͣࢦఆ͢Δ
    ඞཁ࠷௿ݶͷΧϥϜΛ4&-&$5͢Δ
    ҆қʹ4&-&$5
    
    ͠ͳ͍
    FUD "UIFOBΛίετޮ཰ྑ͘׆༻͢ΔͨΊͷϚΠϯυηοτ
35. &0'