Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AWSで始めるビッグデータ分析 Developers.IO 2017

AWSで始めるビッグデータ分析 Developers.IO 2017

Developers.IO 2017登壇資料

Yutaka Yawata

July 01, 2017
Tweet

More Decks by Yutaka Yawata

Other Decks in Technology

Transcript

  1. σʔλ෼ੳͰར༻Ͱ͖Δ"84αʔϏε Amazon
 DynamoDB Amazon
 RDS Amazon 
 Redshift Amazon
 S3

    AWS Snowball AWS Lambda AWS Batch Amazon EC2 AWS DMS Amazon ES Amazon EMR Amazon Kinesis Amazon
 Athena Amazon CloudSearch Amazon QuickSight AWS Data Pipeline Amazon Machine Learning AWS IoT  ଟछଟ༷ͳαʔϏε܈ σʔλҠߦ σʔλϕʔε ετϨʔδ ෼ੳ ݕࡧ *P5 ίϯϐϡʔςΟϯά ػցֶश
  2. ऩू ϑΝΠϧ ετϦʔϜ w ച্σʔλ w ސ٬Ϛελ w ঎඼Ϛελ w

    ߦಈϩά w ΫϦοΫετϦʔϜ w *P5ηϯαʔσʔλ ɏ σʔλλΠϓʹண໨ AWS IoT Amazon S3 Amazon Kinesis Streams Amazon Kinesis Firehose … … … 
  3. ूΊͨσʔλ͸4ʹू໿ Amazon S3 Amazon EC2 AWS Aurora Amazon Redshift AWS

    Lambda Amazon ES Amazon EMR Amazon CloudSearch Amazon QuickSight Amazon Athena AWS IoT Amazon Kinesis Streams Amazon Kinesis Firehose AWS DMS AWS Batch  AWS DynamoDB AWS Lambda AWS Data Pipeline
  4. લॲཧ ϦΞϧλΠϜ όον w ࣌ؒຖɺ೔࣍ɺि࣍ɺ w εϧʔϓοτॏࢹ w ஞ࣍ॲཧ w

    ϨΠςϯγʔॏࢹ Amazon EMR Amazon EC2 AWS Lambda Amazon Kinesis Analytics ɏ ϫʔΫϩʔυʹண໨ ƽ 
  5. อଘɾ෼ੳ σʔλߏ଄ ΞΫηεύλʔϯ อଘ ෼ੳ ݻఆ͞ΕͨεΩʔϚ ෳࡶͳϦϨʔγϣφϧΫΤϦɺ ଟ਺ͷϢʔβʔ͔Βͷಉ࣌ΞΫηε 3%4 େن໛σʔλͷूܭɾநग़

    3FETIJGU 4 3FETIJGU4QFDUSVN εΩʔϚϨε γϯϓϧͳΫΤϦɺ௿͍ϨΠςϯγʔ %ZOBNP%# ੜσʔλʹର͢ΔΞυϗοΫ෼ੳ 4 &.3
 1SFTUP *NQBMB 4QBSL  "UIFOB "UIFOB શจݕࡧ &MBTUJD4FBSDI4FSWJDF $MPVE4FBSDI ɏ σʔλߏ଄ɺΞΫηεύλʔϯʹண໨ 
  6. ՄࢹԽ ఏڙํ๏ ՄࢹԽπʔϧ ηϧϑ#* "NB[PO2VJDL4JHIU ఆܕϨϙʔτ ΧελϜ8FCΞϓϦέʔγϣϯ w %ZOBNP%# -BNCEB

    "1*(BUFXBZ 4 w 3%4 &$
 ϩά෼ੳͷμογϡϘʔυ "NB[PO&MBTUJD4FBSDI4FSWJDF ,JCBOB  ɏ σʔλͷఏڙํ๏ʹண໨͢Δ w "84αʔϏεͷબ୒ࢶ͸গͳ͍αʔυύʔςΟ੡ͷ
 πʔϧ΋ݕ౼ ǟ ǟ
  7. σʔλ෼ੳʹޮ͘"84αʔϏε Amazon S3 Amazon EC2 AWS Aurora Amazon Redshift AWS

    Lambda Amazon ES Amazon EMR Amazon CloudSearch Amazon QuickSight Amazon Athena AWS IoT Amazon Kinesis Streams Amazon Kinesis Firehose AWS DMS AWS Batch  AWS DynamoDB AWS Lambda AWS Data Pipeline ओཁͳαʔϏε ಠஅͱภݟ 
 ͭΛ͝঺հ
  8. "NB[PO3FETIJGUՁ֨ ϊʔυλΠϓ W$16਺ ϝϞϦ༰ྔ ετϨʔδ༰ྔ ϊʔυ਺ Ձ֨
 ϊʔυʗ࣌ؒ *0ੑೳॏࢹ
 44%λΠϓ

    EDMBSHF  (J# 5# ʙ  EDYMBSHF  (J# 5# ʙ  ετϨʔδ༰ྔॏࢹ
 )%%λΠϓ ETYMBSHF  (J# 5# ʙ  ETYMBSHF  (J# 5# ʙ  ˞Ձ֨͸͢΂ͯ౦ژϦʔδϣϯɺ೥݄೔࣌఺ͷ΋ͷͰ͢ɻ  wϊʔυλΠϓຖʹετϨʔδ༰ྔ͕ܾ·͍ͬͯΔ wΫϥελʔͷෳ੡ɺϊʔυλΠϓ΍ϊʔυ਺ͷมߋ͸ൺֱత༰қ
 ˠੑೳݕূΛߦͬͯϊʔυλΠϓΛܾఆ͢Δ
  9. "NB[PO3FETIJGU4QFDUSVN  Amazon S3 w 3FETIJGU͔Β4্ͷϑΝΠϧʹ௚઀ΫΤϦͰ͖Δ
 σʔλϩʔυෆཁ  w ༷ʑͳϑΝΠϧϑΥʔϚοτʹରԠ


    $47ɺ547ɺ1BSRVFU౳ʣ w ΞΫηεස౓͕௿͍σʔλΛ4΁Φϑϩʔυ
 ˠίετ࡟ݮ  w ΫΤϦ୯Ґͷैྔ՝ۚ
 εΩϟϯ͞Εͨσʔλ5#͋ͨΓ 4QFDUSVN૚ $47 547 
 1BSRVFU FUD 3FETIJGU্ͷςʔϒϧͱ δϣΠϯͰ͖Δ SQL
  10. "NB[PO"UIFOB  Amazon S3 w Ϛωʔδυ1SFTUP w αʔόʔϨε w 4্ͷϑΝΠϧʹ௚઀ΫΤϦͰ͖Δ

    w ༷ʑͳϑΝΠϧϑΥʔϚοτΛαϙʔτ
 $47ɺ547ɺ1BSRVFUɺ+40/ɺ"WSP౳ʣ w ΫΤϦ୯Ґͷैྔ՝ۚ
 εΩϟϯ͞Εͨσʔλ5#͋ͨΓ Amazon Athena SQL
  11. "NB[PO,JOFTJT4USFBNT wετϦʔϜσʔλͷऩूͱอଘ w อଘظؒ͸࣌ؒʙ࠷େ೔ؒ wϦΞϧλΠϜॲཧ w σʔλͷೖྗ͔Βऔಘ·Ͱͷ஗Ԇ͸௨ৗඵະຬ wߴ͍εέʔϥϏϦςΟ w εϧʔϓοτ΍σʔλྔʹԠͯ͡εέʔϧΠϯʗΞ΢τՄೳ

    γϟʔυͷ෼ׂɺ౷߹  Amazon Kinesis Streams AWS Lambda Amazon EMR Amazon Kinesis–
 enabled app Amazon Redshift Amazon DynamoDB Amazon S3 Put Put Get Get Get AWS IoT Put
  12. "NB[PO,JOFTJT"OBMZUJDT Amazon Kinesis Streams AWS IoT Put Put Put Amazon

    Kinesis Analytics w ετϦʔϜσʔλʹରͯ͠42-ΫΤϦΛ࣮ߦͰ͖Δ w ,JOFTJT4USFBNT'JSFIPTF͔Β*OQVUˠ,JOFTJT4USFBNT'JSFIPTFʹ0VUQVU w εϧʔϓοτʹԠͯࣗ͡ಈεέʔϧ Amazon Kinesis Firehose Amazon Redshift SQL
  13. ,JOFTJT4USFBNTͱ'JSFIPTFͷൺֱ ػೳ ϨΠςϯγʔ σʔλͷॲཧ εέʔϧΞ΢τʗΠϯ ౦ژϦʔδϣϯͰͷ
 ར༻Մ൱ ,JOFTJT
 4USFBNT ετϦʔϜσʔλͷ


    ऩूͱอଘ ໿ඵ ΧελϜ
 ΞϓϦέʔγϣϯ ϢʔβʔଆͰ
 ίϯτϩʔϧ Մ ,JOFTJT
 'JSFIPTF ετϦʔϜσʔλͷ
 ऩूͱ഑৴ ࠷୹ඵ 4ɺ3FETIJGUɺ
 &MBTUJD4FBSDI 4FSWJDFʹࣗಈ഑৴ ࣗಈ ൱
 όʔδχΞɺΦϨΰϯɺ
 ΞΠϧϥϯυͷΈ  w·ͣ͸,JOFTJT'JSFIPTFΛݕ౼ wϨΠςϯγʔɺσʔλॲཧύλʔϯͰ൑அͯ͠
 ,JOFTJT'JSFIPTFͰࣄ଍Γͳ͍৔߹͸,JOFTJT4USFBNTΛར༻
  14. ,JOFTJTͷՁ֨  Kinesis Streams ߲໨ ྉۚ ౦ژϦʔδϣϯ γϟʔυ࣌ؒ
 .#ඵͷσʔλೖྗͱ.#ඵͷσʔλग़ྗ 

    165ϖΠϩʔυϢχοτ ,# 
   Ϣχοτ͝ͱ  ֦ுσʔλอ࣋ظݶ ࠷େ೔ؒ ɺγϟʔυ࣌ؒ͝ͱ  ˞Ձ֨͸͢΂ͯ೥݄೔࣌఺ͷ΋ͷͰ͢ɻ Kinesis Firehose ߲໨ ྉۚ όʔδχΞϦʔδϣϯ औΓࠐ·Εͨσʔλྔ(#͋ͨΓ ʙ
 ˞5JFSྉۚ  Kinesis Analytics ߲໨ ྉۚ όʔδχΞϦʔδϣϯ ,16 ,JOFTJT1SPDFTTJOH6OJU ɺ࣌ؒ͋ͨΓ 
 ˞5JFSྉۚ 
  15. 2VJDL4JHIUՁ֨ ແྉ࿮ ແྉ࿮
 Ϣʔβʔ 4UBOEBSE&EJUJPO &OUFSQSJTF&EJUJPO αϒεΫϦϓγϣϯ /" ೥ؒ ݄ؒ

    ೥ؒ ݄ؒ Ϣʔβʔʗ݄      όϯυϧ͞Εͨ
 41*$&༰ྔ (# (# (# (# (# ௥Ճͷ41*$&༰ྔ
 (#͋ͨΓʗ݄  (#  ˞Ձ֨͸೥݄೔࣌఺ͷ΋ͷͰ͢ɻ  w Ϣʔβʔ୯Ґͷ՝ۚͱɺར༻͢ΔετϨʔδ 41*$& ༰ྔʹର͢Δ՝ۚ w &OUFSQSJTF&EJUJPOɿ
 "%࿈ܞɺ҉߸Խɺ4".-ϕʔεͷϑΣσϨʔςου440Λαϙʔτ
 ࠓޙػೳ͕௥Ճ͞ΕΔ༧ఆ
  16. ച্෼ੳσʔλม׵ɺ੔ܗ  Amazon
 Athena Amazon
 EMR ੜσʔλ Ճ޻ࡁΈσʔλ Amazon
 EMR

    σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ &.3Ͱσʔλม׵ɺ੔ܗ ؆қͳॲཧͰ͋Ε͹&$Ͱࡁ·ͤΔ͜ͱ΋ ǟ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ)
  17. ച্෼ੳ3FETIJGU΁औࠐΈ  Amazon
 Athena Amazon
 EMR ੜσʔλ Ճ޻ࡁΈσʔλ Amazon Redshift

    όοναʔόʔ Amazon
 EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ όοναʔόʔͰ
 $01:ίϚϯυΛ࣮ߦ ǟ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ)
  18. ച্෼ੳ3FETIJGU΁औࠐΈ  Amazon
 Athena Amazon
 EMR ੜσʔλ Ճ޻ࡁΈσʔλ Amazon Redshift

    όοναʔόʔ Amazon
 EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ σΟϝϯγϣϯ ςʔϒϧ͸ 3FETIJGU΁ 4QFDUSVN ϑΝΫτςʔϒϧ͸4ʹɻ 4QFDUSVNͰΞΫηε ǟ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ) Redshift Spectrum͕౦ژϦʔδϣϯͰ࢖͑ΔΑ͏ʹͳͬͨΒɺɺɺ
 (·ͨ͸όʔδχΞʗΦϨΰϯϦʔδϣϯͰ෼ੳج൫Λߏங͢ΔͳΒɺɺɺ)
  19. ച্෼ੳ#*πʔϧͰ෼ੳɺՄࢹԽ  Amazon
 Athena Amazon
 EMR ੜσʔλ Ճ޻ࡁΈσʔλ Amazon QuickSight

    Amazon Redshift όοναʔόʔ Amazon
 EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ 4QFDUSVN 2VJDL4JHIUͰՄࢹԽɻ ͦͷଞ#*πʔϧ΋࢖༻Մೳ ՄࢹԽ ǟ Internet ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ) (όʔδχΞ or ΦϨΰϯ)
  20. ച্෼ੳϨϙʔτఏڙ  Amazon
 Athena Amazon
 EMR ੜσʔλ Ճ޻ࡁΈσʔλ Amazon Redshift

    όοναʔόʔ Amazon
 EMR 4QFDUSVN Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FEWJFX 
 Ϩϙʔτදࣔ
 αʔόʔ ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ) σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ ϨϙʔτӾཡऀ (100ਓʙ) Amazon QuickSight ǟ ՄࢹԽ Internet Internet (όʔδχΞ or ΦϨΰϯ) σʔλϚʔτΛ3%4ʹอଘ͠ɺ ϨϙʔτӾཡऀʹ։์
  21. ച্෼ੳશମ૾  Amazon
 Athena Amazon
 EMR ੜσʔλ Ճ޻ࡁΈσʔλ Amazon Redshift

    όοναʔόʔ Amazon
 EMR 4QFDUSVN Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FEWJFX 
 Ϩϙʔτදࣔ
 αʔόʔ Internet σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ Internet ΞυϗοΫ෼ੳ σʔλม׵ɺ੔ܗ Amazon QuickSight ǟ ϨϙʔτӾཡऀ (100ਓʙ) ՄࢹԽ ෼ੳ୲౰ऀ (਺ਓʙे਺ਓ) (όʔδχΞ or ΦϨΰϯ)
  22. ϩά෼ੳσʔλऩू  Amazon Kinesis Firehose ߦಈϩά ෼ੳ୲౰ऀ ౦ژϦʔδϣϯ όʔδχΞ or

    ΦϨΰϯϦʔδϣϯ ,JOFTJT'JSFIPTF΁
 ϩάσʔλΛ1VU Web/App
 αʔόʔ
  23. ϩά෼ੳ3FETIJGUͱ4΁഑৴  ੜσʔλ Amazon Redshift Web/App
 αʔόʔ Amazon Kinesis Firehose

    Lambda function ߦಈϩά ෼ੳ୲౰ऀ ౦ژϦʔδϣϯ όʔδχΞ or ΦϨΰϯϦʔδϣϯ ϩάσʔλΛ 3FETIJGUͱ4ʹ഑৴
  24. ϩά෼ੳ#*πʔϧͰ෼ੳɺՄࢹԽ  ੜσʔλ Amazon Redshift Amazon QuickSight Amazon Kinesis Firehose

    Lambda function ߦಈϩά ෼ੳ୲౰ऀ Internet ౦ژϦʔδϣϯ όʔδχΞ or ΦϨΰϯϦʔδϣϯ 2VJDL4JHIUͰՄࢹԽɻ ͦͷଞ#*πʔϧ΋࢖༻Մೳ Web/App
 αʔόʔ
  25. ϩά෼ੳੜσʔλͷΞυϗοΫ෼ੳ  Amazon
 Athena ੜσʔλ Amazon Redshift Amazon QuickSight Amazon

    Kinesis Firehose Lambda function ߦಈཤྺ ෼ੳ୲౰ऀ Internet ౦ژϦʔδϣϯ όʔδχΞ or ΦϨΰϯϦʔδϣϯ "UIFOBͰ
 ΞυϗοΫ෼ੳ Web/App
 αʔόʔ
  26. ϩά෼ੳσʔλऩू  Amazon
 Athena ੜσʔλ Amazon Redshift Amazon QuickSight Amazon

    Kinesis Firehose Lambda function ߦಈϩά ΞΫηεϩά Amazon Kinesis Firehose Lambda function ෼ੳ୲౰ऀ γεςϜ
 ؅ཧऀ Internet ౦ژϦʔδϣϯ όʔδχΞ or ΦϨΰϯϦʔδϣϯ ,JOFTJT'JSFIPTF΁
 ϩάσʔλΛ1VU Web/App
 αʔόʔ
  27. ϩά෼ੳ&MBTUJD4FBSDIͰՄࢹԽ  Amazon
 Athena ߦಈϩά
 ੜσʔλ Amazon Redshift Amazon QuickSight

    Amazon Kinesis Firehose Amazon ES Lambda function ߦಈϩά ΞΫηεϩά Amazon Kinesis Firehose ſ Lambda function ෼ੳ୲౰ऀ γεςϜ
 ؅ཧऀ ౦ژϦʔδϣϯ όʔδχΞ or ΦϨΰϯϦʔδϣϯ Internet Internet ϩάσʔλΛ
 &MBTUJD4FBSDI4FSWJDFʹ഑৴ɻ
 ,JCBOBͰՄࢹԽ Web/App
 αʔόʔ ΞΫηεϩά ੜσʔλ ੜσʔλ͸4ʹอଘ͓ͯ͘͠
  28. ϩά෼ੳϦΞϧλΠϜ௨஌  Amazon
 Athena ੜσʔλ Amazon Redshift Amazon QuickSight Amazon

    Kinesis Streams Amazon Kinesis Analytics Amazon Kinesis Firehose Amazon ES Lambda function ߦಈϩά ΞΫηεϩά Amazon Kinesis Firehose ſ Lambda function Lambda function ෼ੳ୲౰ऀ γεςϜ
 ؅ཧऀ ౦ژϦʔδϣϯ όʔδχΞ or ΦϨΰϯϦʔδϣϯ Internet Internet ,JOFTJT4USFBNTͷΠϕϯττϦΨʔͰ
 -BNCEBϑΝϯΫγϣϯΛ࣮ߦ͠
 νϟοτʹ௨஌ Web/App
 αʔόʔ ΞΫηεϩά ੜσʔλ ,JOFTJT"OBMZUJDTͰ
 σʔλநग़ˍ,JOFTJT4USFBNTʹ഑৴
  29. ϩά෼ੳશମ૾  Amazon
 Athena Amazon Redshift Amazon QuickSight Amazon Kinesis

    Streams Amazon Kinesis Analytics Amazon Kinesis Firehose Amazon ES Lambda function ߦಈϩά ΞΫηεϩά Amazon Kinesis Firehose ſ Lambda function Lambda function ෼ੳ୲౰ऀ γεςϜ
 ؅ཧऀ ౦ژϦʔδϣϯ όʔδχΞ or ΦϨΰϯϦʔδϣϯ Internet Internet Web/App
 αʔόʔ ΞΫηεϩά ੜσʔλ ߦಈϩά
 ੜσʔλ