AWSをフル活用した大規模ログ収集基盤 / log collecting foundation of big data by AWS

SoftBank × AWS Drink Meetup #1

Transcript

1. "84Λϑϧ׆༻ͨ͠
   େن໛ϩάऩूج൫ 
   4PGUCBOL
   $PSQ
2. ᖛాɹ༎ ιϑτόϯΫגࣜձࣾ
   ݉ "HPPQ
   ޷͖ͳ"84αʔϏε "UIFOB
   
   (MVF όοΫΤϯυશൠ
3. Ґஔ৘ใऩू ม׵ ׆༻
4. Ґஔ৘ใऩू ม׵ ׆༻
5. 
    εϚϗΞϓϦͷاը։ൃ اը͔Β։ൃ·Ͱશͯ಺੡Խ ϩάΛूΊΔػೳΛΞϓϦʹ௥Ճ ϏοάσʔλͷऩूɾՁ஋ͷఏڙ
8. Ґஔ৘ใऩू ม׵ ׆༻
10. Title Text Main Point -! - 10 ϐϯϙΠϯτਓྲྀσʔλͷՄࢹԽ
11. ڵຯ͕͋Δํ͸ฐࣾ)1·Ͱ IUUQTXXXBHPPQDPKQ
12. Ґஔ৘ใऩू ม׵ ਓͷྲྀΕ
13. ϩάσʔλͷ֓ཁ
    εϚʔτϑΥϯΞϓϦ͔Βऩू
    த਎͸௨৴ঢ়گͳͲʹؔ͢Δϩά
    +40/ܗࣜ
    ೔Ͱऩू͢ΔσʔλϘϦϡʔϜ
    ݅਺
    ਺ԯ݅
    σʔλαΠζ
    5#௒ γεςϜ͕ऩू͢Δϩά
14. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF
15. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF +FOLJOT
    PO
    &$ -BNCEB %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF ऩूɾ஝ੵ
16. ऩूɾ஝ੵ +BWB൛ͷ,JOFTJT
    1SPEVDFS
    -JCSBSZͱ,JOFTJT
    $POTVNFS
    -JCSBSZΛར༻
    ,JOFTJTϓϩσϡʔαʔ͸ड৴ͨ͠+40/Λ΄΅ͦͷ··,JOFTJTʹ౤ೖ
    ,JOFTJTʹೖΕ͓͚ͯ͹࣌ؒ
    ࠷௕೔ؒ
    σʔλϩετͷ৺഑͕ແ͍ͨΊ
    ,JOFTJTίϯγϡʔϚʔ͸ҎԼͷॲཧΛ࣮ࢪ
    +40/Λύʔεͯ͠ޙଓͷ෼ੳͰ࢖༻͢Δ৘ใΛ෇Ճ
    &$಺ͰՔಇ͢ΔqVFOUEʹόοϑΝͯ͠ҰఆִؒͰ4ʹ([JQܗࣜͰग़ྗ
    <1PJOU>
    ,JOFTJTಋೖͰ҆ఆੑ͕͔ͳΓ޲্ˠσʔλऩूܥʹ͸͔ͳΓΦεεϝ "VUP4DBMF
    &$ ,JOFTJT "VUP4DBMF
    &$ 4
17. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF &5-
18. &5-
    3FETIJGUɾ)JWFϩʔυ <1PJOU>
    )JWFϝλετΞ͸3%4ΑΓ΋(MVF
    %BUB
    $BUBMPH͕Φεεϝ
    3%4͕410'
    ୯Ұো֐఺
    ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋Δ
    3%4͸ಉ࣌઀ଓ਺ͷ੍ݶ͕͋ΔͨΊ&.3Ϋϥελʔͷಉ࣌ىಈ਺ʹ੍ݶ͕ੜ͡Δ
    ಉ࣌઀ଓ਺͸ϝϞϦґଘͷύϥϝʔλʔ
    ಉ࣌઀ଓ਺ͷ্ݶΛ্͛ΔͨΊʹ͸ΠϯελϯελΠϓΛ্͛Δඞཁ͕͋Δ
    ϝλσʔλ͔͠؅ཧ͠ͳ͍ͷʹϦονͳεϖοΫ͸໪ମແ͍
    (MVF
    %BUB
    $BUBMPH͸ϑϧϚωʔδυ͔ͭಉ࣌઀ଓ਺ͷ੍ݶͳ͠ 4QBSL
    PO
    &.3 4
    ([JQ 4
    1BSRVFU 3FETIJGU 3%4
    )JWF
    .FUBTUPSF ऩूɾ஝ੵ 4
19. &5-
    3FETIJGUɾ)JWFϩʔυ 4QBSL
    PO
    &.3Ͱ͸࣌ؒ෼ͷϩάΛೖྗͱͯ͠ҎԼεςοϓΛ࣮ߦ
    4%JTU$QͰ໿ ݸ
    ,#ڃ
    ͷϑΝΠϧΛ໿ݸ
    .#d(#ڃ
    ͷϑΝΠϧʹू໿
    <1PJOU>
    4QBSL
    )BEPPQ
    ͸େྔͷখ͞ͳϑΝΠϧΛѻ͏ͷ͸ෆಘҙ
    4QBSLͰ([JQQFE
    +40/Λग़ྗ
    
    )JWFςʔϒϧʹ*/4&35
    07&383*5&
    εςοϓऴྃޙɺ&.3Ϋϥελʔ͸ࣗಈ࡟আ
    ࣌ؒ෼ͷॲཧΛಠཱͨ͠ΫϥελʔͰ࣮ߦ͢ΔͷͰϦΧόϦʔͳͲ͕΍Γ΍͍͢
    4্ͷ([JQϑΝΠϧΛ3FETIJGUͷ$01:ίϚϯυͰϩʔυ 4QBSL
    PO
    &.3 4
    ([JQ 4
    1BSRVFU 3FETIJGU 3%4
    )JWF
    .FUBTUPSF ऩूɾ஝ੵ 4
20. γεςϜΞʔΩςΫνϟ "VUP4DBMJOH
    &$ ,JOFTJT "VUP4DBMJOH
    &$ 4 4 3FETIJGU "UIFOB 1SFTUP
    PO
    &.3 4 4QBSL
    PO
    &.3 (MVF +FOLJOT
    PO
    &$ -BNCEB %FWFMPQFST
    %BUB
    "OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4
    )JWF
    .FUBTUPSF ෼ੳ
21. ෼ੳπʔϧ "UIFOB 1SFTUP
    PO
    &.3 (MVF %FWFMPQFST
    %BUB
    "OBMZTUT 3FETIJGU "UIFOB
    1SFTUP
    PO
    &.3
    ೥ࠒΑΓར༻։࢝
    ɹ
    3FETIJGU
    ϩάऩू։࢝౰ॳΑΓར༻
    ࠓޙͷϩά૿ՃΛݟࠐΈɺ
    &.3
    "UIFOB΁Ҡߦத
    4 "UIFOBͷར༻ྉۚߴಅ
    
22. "UIFOBͷར༻ྉۚߴಅͷݪҼ 3FEBTIͷΫΤϦεέδϡʔϧػೳ
    3FEBTIʹอଘͨ͠ΫΤϦΛࢦఆͨ͠ස౓Ͱ࣮ߦͰ͖Δ "UIFOBͷ࢖͍উखͷྑ͞
    ͋Δఔ౓ɺແ஡ͳΫΤϦ΋ݱ࣮తͳ࣌ؒͰॲཧͰ͖Δ ਺5#ΛεΩϟϯ͢ΔΫΤϦ܈͕සൟʹ࣮ߦ
    τʔλϧεΩϟϯαΠζ͕1#ΛΦʔόʔ
23. "UIFOBར༻ͷڭ܇ ίετҙࣝ
    ͜ͷΫΤϦΛ࣮ߦ͢Δ͜ͱͰԿ(#εΩϟϯ͢Δͷ͔ɺͦͷ݁Ռྉۚ͸͍͘Βֻ ͔Δ͔Λҙࣝ͢Δ͜ͱ͸ඇৗʹॏཁ
    "84؅ཧऀ޲͚
    σʔλΞφϦετʹྉۚΛೝࣝͤ͞Δ࢓૊Έ͕ඞཁ
    %3:ݪଇ
    %PO`U
    3FQFBU
    :PVSTFMG
    ϔϏʔͳΫΤϦΛԿճ΋࣮ߦ͠ͳ͍ɾͤ͞ͳ͍
    Α͋͘ΔΫΤϦ͸ճ͚࣮ͩߦͯ݁͠ՌΛอଘ͢Δ
    ͓࡞๏ͷపఈ
    εΩϟϯྔ௿ݮͷͨΊύʔςΟγϣϯΛඞͣࢦఆ͢Δ
    ඞཁ࠷௿ݶͷΧϥϜΛ4&-&$5͢Δ
    4&-&$5
    
    ͸μϝ
    FUD "UIFOBഁ࢈͠ͳ͍ͨΊʹϢʔβʔ͕࣋ͭ΂͖ϚΠϯυηοτ
24. ࠓޙͷల๬ SFJOWFOUͰൃද͞Εͨ৽αʔϏεΛੵۃతʹಋೖ༧ఆ 5JNFTUSFBN
    $POUPM
    5PXFS
    -BLF
    'PSNBUJPO
    5SBOTJU
    (BUFXBZ
25. &0'