AWSで始めるビッグデータ分析 Developers.IO 2017 WORLD in 名古屋

Developers.IO 2017 WORLD in 名古屋 登壇資料

Transcript

1. "84Ͱ࢝ΊΔϏοάσʔλ෼ੳ %FWFMPQFST*0
   
   803-%
   JO
   ໊ݹ԰ ೥݄೔
   Ϋϥεϝιουגࣜձࣾ
   ീ഼๛ DNEFWJP
2. ࣗݾ঺հ wീ഼๛ʢ:VUBLB
   :BXBUBʣ
   wϓϦηʔϧε୲౰
   wݩ&31ύοέʔδͷج൫ίϯαϧ ࣗݾ঺հ 
3. ͜ͷηογϣϯʹ͍ͭͯ
   ύΠϓϥΠϯͰߟ͑Δσʔλ෼ੳج൫ɹ
   σʔλ෼ੳϦϑΝϨϯεΞʔΩςΫνϟ
   
4. ηογϣϯͷର৅ऀ w"84Ͱσʔλ෼ੳΛ࢝Ί͍ͨ
   w"84Ͱσʔλ෼ੳج൫Λߏங͢ΔͨΊͷ
 ϕετϓϥΫςΟεΛ஌Γ͍ͨ
   w"84ͷར༻ܦݧ͕͋Δ 71$
   &$
   3%4
   4
   FUD 
5. ͜ͷηογϣϯͰ͓࿩͠͠ͳ͍͜ͱ wͳͥ"84͔ʁ
   w֤"84αʔϏεͷৄࡉͳػೳ΍εϖοΫ
   wػցֶशɺ"* 
6. ͜ͷηογϣϯʹ͍ͭͯ
   ύΠϓϥΠϯͰߟ͑Δσʔλ෼ੳج൫ɹ
   σʔλ෼ੳϦϑΝϨϯεΞʔΩςΫνϟ
   
7. ༷ʑͳϏοάσʔλ ɨ  104σʔλ ϞόΠϧΞϓϦ ߦಈϩά ΫϦοΫ
 ετϦʔϜ  4/4 ηϯλʔσʔλ  ɓ
8. σʔλ෼ੳͰར༻Ͱ͖Δ"84αʔϏε Amazon
 DynamoDB Amazon
 RDS Amazon 
 Redshift Amazon
 S3 AWS Snowball AWS Lambda AWS Batch Amazon EC2 AWS DMS Amazon ES Amazon EMR Amazon Kinesis Amazon
 Athena Amazon CloudSearch Amazon QuickSight AWS Data Pipeline Amazon Machine Learning AWS IoT  ଟछଟ༷ͳαʔϏε܈ σʔλҠߦ σʔλϕʔε ετϨʔδ ෼ੳ ݕࡧ *P5 ίϯϐϡʔςΟϯά ػցֶश AWS Glue
9. ͲͷαʔϏεΛ࢖͏΂͖͔ʁ  Ͳ͏΍ͬͯ"84ʹσʔλΛΞοϓϩʔυ͢Δʁ σʔλͷՃ޻͸Ͳ͜Ͱ΍Δʁ&$ʁ 3FETIJGU࢖ͬͨํ͕Α͍ʁ3%4Ͱ΋େৎ෉ʁ
10. σʔλ෼ੳΞʔΩςΫνϟͷݪଇ  wσʔλ෼ੳΛύΠϓϥΠϯͰߟ͑Δ
    w ऩूɺલॲཧɺอଘɺ෼ੳɺՄࢹԽ
    w"84ͷ֤αʔϏε͕Կʹ࠷దԽ͞Ε͍ͯΔ͔ʁ·ͨ
 ࠷దԽ͞Εͨ݁ՌͷτϨʔυΦϑ͸Կ͔ʁΛཧղ͢Δ
    w εϧʔϓοτɺϨΠςϯγʔɺεέʔϥϏϦςΟɺ
 ੔߹ੑɺѻ͑ΔσʔλλΠϓʗσʔλߏ଄ɺ
 ΧελϚΠζੑɺίετɺ্ݶ஋
    w֤αʔϏεͷಛ௃Λཧղ্ͨ͠ͰɺύΠϓϥΠϯ্ͷ
 ͦΕͧΕͷδϣϒʹ࠷దͳαʔϏεΛબ୒͢Δ
11. ϏοάσʔλύΠϓϥΠϯ લॲཧ ऩू อଘ ෼ੳ ՄࢹԽ |   
12. ϏοάσʔλύΠϓϥΠϯ
    
    อଘ લॲཧ ऩू อଘ ෼ੳ ՄࢹԽ |  
13. σʔλ͸4ʹू໿ Amazon S3 Amazon EC2 AWS Aurora Amazon Redshift AWS Lambda Amazon ES Amazon EMR Amazon CloudSearch Amazon QuickSight Amazon Athena AWS IoT Amazon Kinesis Streams Amazon Kinesis Firehose AWS DMS AWS Batch  AWS DynamoDB
14. ϏοάσʔλύΠϓϥΠϯ
    
    ऩू લॲཧ ऩू อଘ ෼ੳ ՄࢹԽ |   
15. ऩू ϑΝΠϧ ετϦʔϜ w ച্σʔλ
    w ސ٬Ϛελ
    w ঎඼Ϛελ w ߦಈϩά
    w ΫϦοΫετϦʔϜ
    w *P5ηϯαʔσʔλ ɏ σʔλλΠϓʹண໨ AWS IoT Amazon S3 Amazon Kinesis Streams Amazon Kinesis Firehose … … … 
16. ϏοάσʔλύΠϓϥΠϯ
    
    લॲཧ લॲཧ ऩू อଘ ෼ੳ ՄࢹԽ |   
17. લॲཧ ϦΞϧλΠϜ όον w ࣌ؒຖɺ೔࣍ɺि࣍ɺ
    w εϧʔϓοτॏࢹ w ஞ࣍ॲཧ
    w ϨΠςϯγʔॏࢹ Amazon EMR Amazon EC2 AWS Lambda Amazon Kinesis Analytics ɏ ϫʔΫϩʔυʹண໨ ƽ 
18. ϏοάσʔλύΠϓϥΠϯ
    
    ෼ੳ લॲཧ ऩू อଘ ෼ੳ ՄࢹԽ |   
19. ෼ੳ σʔλߏ଄ ΞΫηεύλʔϯ σʔλετΞ ෼ੳ ݻఆ͞ΕͨεΩʔϚ ෳࡶͳϦϨʔγϣφϧΫΤϦɺ
    ଟ਺ͷϢʔβʔ͔Βͷಉ࣌ΞΫηε 3%4 େن໛σʔλͷूܭɾநग़ 3FETIJGU 4 3FETIJGU
    4QFDUSVN εΩʔϚϨε γϯϓϧͳΫΤϦɺ௿͍ϨΠςϯγʔ %ZOBNP%# ੜσʔλʹର͢ΔΞυϗοΫ෼ੳ 4 &.3
 1SFTUP
    *NQBMB
    4QBSL
    "UIFOB "UIFOB શจݕࡧ &MBTUJD4FBSDI
    4FSWJDF
    $MPVE4FBSDI ɏ σʔλߏ଄ɺΞΫηεύλʔϯʹண໨ 
20. ϏοάσʔλύΠϓϥΠϯ
    
    ՄࢹԽ લॲཧ ऩू อଘ ෼ੳ ՄࢹԽ |   
21. ՄࢹԽ ఏڙํ๏ ՄࢹԽπʔϧ ηϧϑ#* "NB[PO
    2VJDL4JHIU ఆܕϨϙʔτ ΧελϜ8FCΞϓϦέʔγϣϯ
    w %ZOBNP%#
    
    -BNCEB
    
    "1*
    (BUFXBZ
    
    4
    w 3%4
    
    &$
 ϩά෼ੳͷμογϡϘʔυ "NB[PO
    &MBTUJD4FBSDI
    4FSWJDF ,JCBOB  ɏ σʔλͷఏڙํ๏ʹண໨͢Δ w "84αʔϏεͷબ୒ࢶ͸গͳ͍αʔυύʔςΟ੡ͷ
 πʔϧ΋ݕ౼
    ǟ
    ǟ
22. ͜ͷηογϣϯʹ͍ͭͯ
    ύΠϓϥΠϯͰߟ͑Δσʔλ෼ੳج൫ɹ
    σʔλ෼ੳϦϑΝϨϯεΞʔΩςΫνϟ
     
23. ϦϑΝϨϯεΞʔΩςΫνϟ  wǡ ച্෼ੳ
    wΦϯϓϨϛεͷ%#ʹ͋Δച্σʔλ
    wҎԼͷ෼ੳ؀ڥΛఏڙ
    wੜσʔλʹର͢ΔΞυϗοΫ෼ੳ
    wηϧϑ#*
    wఆܕϨϙʔτ
24. ച্෼ੳ
    
    σʔλऩू  ੜσʔλ σʔλ͸4ʹू໿ σʔλΞοϓϩʔυ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ)
25. ച্෼ੳ
    
    ੜσʔλͷΞυϗοΫ෼ੳ  Amazon
 Athena ੜσʔλ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ "UIFOBͰ
 ΞυϗοΫ෼ੳ ǟ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ)
26. ച্෼ੳ
    
    σʔλม׵ɺ੔ܗ  Amazon
 Athena ੜσʔλ Ճ޻ࡁΈσʔλ Amazon
 EMR σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ &.3Ͱσʔλม׵ɺ੔ܗ
    ؆қͳॲཧͰ͋Ε͹&$Ͱࡁ·ͤΔ͜ͱ΋ ǟ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ)
27. ച্෼ੳ
    
    3FETIJGU΁औࠐΈ  Amazon
 Athena ੜσʔλ Ճ޻ࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
 EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ όοναʔόʔͰ
 $01:ίϚϯυΛ࣮ߦ ǟ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ)
28. ച্෼ੳ
    
    3FETIJGU΁औࠐΈ  Amazon
 Athena ੜσʔλ Ճ޻ࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
 EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ σΟϝϯγϣϯ
    ςʔϒϧ͸
    3FETIJGU΁ 4QFDUSVN
    ϑΝΫτςʔϒϧ͸4ʹɻ
    4QFDUSVNͰΞΫηε ǟ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ)
29. ച্෼ੳ
    
    #*πʔϧͰ෼ੳɺՄࢹԽ  Amazon
 Athena ੜσʔλ Ճ޻ࡁΈσʔλ Amazon QuickSight Amazon Redshift όοναʔόʔ Amazon
 EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ 4QFDUSVN
    2VJDL4JHIUͰՄࢹԽɻ
    ͦͷଞ#*πʔϧ΋࢖༻Մೳ ՄࢹԽ ǟ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ)
30. ച্෼ੳ
    
    Ϩϙʔτఏڙ  Amazon
 Athena ੜσʔλ Ճ޻ࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
 EMR 4QFDUSVN
    Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FE
    WJFX 
 Ϩϙʔτදࣔ
 αʔόʔ ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ) σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ ϨϙʔτӾཡऀ (100ਓʙ) Amazon QuickSight ǟ ՄࢹԽ Internet Internet σʔλϚʔτΛ3%4ʹอଘ͠ɺ
    ϨϙʔτӾཡऀʹ։์
31. ച্෼ੳ
    
    શମ૾  Amazon
 Athena ੜσʔλ Ճ޻ࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
 EMR 4QFDUSVN
    Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FE
    WJFX 
 Ϩϙʔτදࣔ
 αʔόʔ ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ) σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ ϨϙʔτӾཡऀ (100ਓʙ) Amazon QuickSight ǟ ՄࢹԽ Internet Internet
32. ϦϑΝϨϯεΞʔΩςΫνϟ  w | 8FC౤ථγεςϜ
    w8FC͔Β౤ථσʔλΛड͚෇͚
    w౤ථσʔλΛɺ
    wϦΞϧλΠϜʹूܭ
    wޙ೔#*πʔϧͰ෼ੳ
33. 8FC౤ථγεςϜ
    
    ౤ථσʔλड෇  ୲౰ऀ Amazon Kinesis Streams Internet ౤ථσʔλड෇ Internet "1*
    (BUFXBZΛϓϩΩγʹͯ͠
    ,JOFTJT
    4USFBNʹ౤ථσʔλΛొ࿥ Amazon API Gateway Amazon CloudFront ౤ථϖʔδ͸4
    
    $MPVE'SPOUͰ഑৴ Amazon S3 ౤ථը໘
34. 8FC౤ථγεςϜ
    
    %#΁ͷొ࿥  ୲౰ऀ Amazon Kinesis Streams Amazon
 DynamoDB AWS Lambda ౤ථσʔλड෇ ౤ථσʔλొ࿥ Internet ౤ථσʔλ -BNCEBͰ%ZOBNP%#ʹ
 ౤ථσʔλΛొ࿥ Internet Amazon CloudFront ౤ථσʔλड෇ Amazon S3 Amazon API Gateway ౤ථը໘
35. 8FC౤ථγεςϜ
    
    ϦΞϧλΠϜूܭ  ୲౰ऀ Amazon Kinesis Streams Amazon Kinesis Analytics Amazon Kinesis Streams AWS Lambda Amazon
 DynamoDB AWS Lambda ౤ථσʔλड෇ ౤ථσʔλूܭ αϚϦσʔλొ࿥ ౤ථσʔλొ࿥ Internet αϚϦσʔλ ౤ථσʔλ ूܭͨ͠αϚϦσʔλΛ
    %ZOBNP%#ʹొ࿥ Internet Amazon CloudFront Amazon API Gateway Amazon S3 ౤ථը໘ ,JOFTJT
    "OBMZUJDTͰ
 ౤ථσʔλΛϦΞϧλΠϜूܭ
36. 8FC౤ථγεςϜ
    
    ΞʔΧΠϒ  ୲౰ऀ Amazon Kinesis Streams Amazon
 DynamoDB AWS Lambda ౤ථσʔλड෇ ౤ථσʔλొ࿥ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ౤ථσʔλ Internet Amazon CloudFront Amazon API Gateway Amazon S3 ౤ථը໘ શ౤ථσʔλ͸,JOFTJT
    'JSFIPTFܦ༝Ͱ
    4ʹΞʔΧΠϒ Amazon Kinesis Analytics Amazon Kinesis Streams ౤ථσʔλूܭ AWS Lambda αϚϦσʔλొ࿥
37. 8FC౤ථγεςϜ
    
    σʔλࢀর  ୲౰ऀ ౦ژϦʔδϣϯ Amazon Kinesis Streams Amazon
 DynamoDB AWS Lambda ౤ථσʔλड෇ ౤ථσʔλొ࿥ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ౤ථσʔλ AWS Lambda %ZOBNP%#
 4USFBN ౤ථσʔλΛϦΞϧλΠϜʹࢀর Internet Amazon API Gateway Amazon CloudFront ूܭ݁Ռ͸4ʹΞοϓϩʔυ͠
    $MPVE'SPOU͔Β഑৴ Amazon S3 ౤ථը໘ Amazon Kinesis Analytics Amazon Kinesis Streams ౤ථσʔλूܭ AWS Lambda αϚϦσʔλొ࿥ Amazon S3 ౤ථ؅ཧը໘
38. 8FC౤ථγεςϜ
    
    શମ૾  ୲౰ऀ ౦ژϦʔδϣϯ Amazon Kinesis Streams Amazon
 DynamoDB AWS Lambda ౤ථσʔλड෇ ౤ථσʔλొ࿥ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ౤ථσʔλ AWS Lambda %ZOBNP%#
 4USFBN Internet Amazon API Gateway Amazon CloudFront Amazon S3 ౤ථ؅ཧը໘ Amazon S3 ౤ථը໘ Amazon Kinesis Analytics Amazon Kinesis Streams ౤ථσʔλूܭ AWS Lambda αϚϦσʔλొ࿥
39. ·ͱΊ  wσʔλ෼ੳΛύΠϓϥΠϯͰߟ͑Δ
    w"84ͷ֤αʔϏε͕Կʹ࠷దԽ͞Ε͍ͯΔ͔ʁ·ͨ
 ࠷దԽ͞Εͨ݁ՌͷτϨʔυΦϑ͸Կ͔ʁΛཧղ͢Δ
    w֤αʔϏεͷಛ௃Λཧղ্ͨ͠ͰɺύΠϓϥΠϯ্ͷ
 ͦΕͧΕͷδϣϒʹ࠷దͳαʔϏεΛબ୒͢Δ
40. ׀幠耮֮׶ָהֲ׀ְׂת׃׋կ