Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AWSで始めるビッグデータ分析 Developers.IO 2017 WORLD in 名古屋
Search
Yutaka Yawata
September 28, 2017
Technology
1
140
AWSで始めるビッグデータ分析 Developers.IO 2017 WORLD in 名古屋
Developers.IO 2017 WORLD in 名古屋 登壇資料
Yutaka Yawata
September 28, 2017
Tweet
Share
More Decks by Yutaka Yawata
See All by Yutaka Yawata
AWSで始めるビッグデータ分析 Developers.IO 2017
yuyawata
1
2.9k
AWSで作るオムニチャネル戦略実現のためのマーケティング・プラットフォーム
yuyawata
1
1.6k
Other Decks in Technology
See All in Technology
データベース02: データベースの概念
trycycle
0
160
現代CSSフレームワークの内部実装とその仕組み
poteboy
7
3.6k
どうするコスト最適化のトレードオフ
tetsuyaooooo
1
530
AOAI をきっかけに 社内の Azure 管理を見直した話
recruitengineers
PRO
1
290
アクセシビリティを考慮したUI/CSSフレームワーク・ライブラリ選定
yajihum
2
1k
私が trocco を推す理由
__allllllllez__
1
240
Janus
bkuhlmann
1
490
Cracking the KubeCon CfP
inductor
2
250
長期間TiDBを使ってきた話 @ 私たちはなぜNewSQLを使うのかTiDB選定5社が語る選定理由と活用LT / Experiences with TiDB Over Time
chibiegg
2
900
生成AIの変革の時代に、直近1年で直面した課題とその解決策
ktc_wada
0
290
LLM開発・活用の舞台裏@2024.04.25
yushin_n
1
320
元インフラエンジニアに成る / Human Resources to Human Relations
bobtani
4
920
Featured
See All Featured
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
StorybookのUI Testing Handbookを読んだ
zakiyama
13
4.6k
Typedesign – Prime Four
hannesfritz
36
2.1k
Web Components: a chance to create the future
zenorocha
305
41k
Agile that works and the tools we love
rasmusluckow
325
20k
Done Done
chrislema
178
15k
Into the Great Unknown - MozCon
thekraken
10
990
Embracing the Ebb and Flow
colly
80
4.1k
The Art of Programming - Codeland 2020
erikaheidi
42
12k
The Language of Interfaces
destraynor
151
23k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
14
1.5k
A Philosophy of Restraint
colly
197
16k
Transcript
"84Ͱ࢝ΊΔϏοάσʔλੳ %FWFMPQFST*0803-%JO໊ݹ ݄ Ϋϥεϝιουגࣜձࣾീ഼๛ DNEFWJP
ࣗݾհ wീ഼๛ʢ:VUBLB:BXBUBʣ wϓϦηʔϧε୲ wݩ&31ύοέʔδͷج൫ίϯαϧ ࣗݾհ
͜ͷηογϣϯʹ͍ͭͯ ύΠϓϥΠϯͰߟ͑Δσʔλੳج൫ɹ σʔλੳϦϑΝϨϯεΞʔΩςΫνϟ
ηογϣϯͷରऀ w"84ͰσʔλੳΛ࢝Ί͍ͨ w"84Ͱσʔλੳج൫Λߏங͢ΔͨΊͷ ϕετϓϥΫςΟεΛΓ͍ͨ w"84ͷར༻ܦݧ͕͋Δ 71$ &$ 3%4 4 FUD
͜ͷηογϣϯͰ͓͠͠ͳ͍͜ͱ wͳͥ"84͔ʁ w֤"84αʔϏεͷৄࡉͳػೳεϖοΫ wػցֶशɺ"*
͜ͷηογϣϯʹ͍ͭͯ ύΠϓϥΠϯͰߟ͑Δσʔλੳج൫ɹ σʔλੳϦϑΝϨϯεΞʔΩςΫνϟ
༷ʑͳϏοάσʔλ ɨ 104σʔλ ϞόΠϧΞϓϦ ߦಈϩά ΫϦοΫ ετϦʔϜ 4/4
ηϯλʔσʔλ ɓ
σʔλੳͰར༻Ͱ͖Δ"84αʔϏε Amazon DynamoDB Amazon RDS Amazon Redshift Amazon S3
AWS Snowball AWS Lambda AWS Batch Amazon EC2 AWS DMS Amazon ES Amazon EMR Amazon Kinesis Amazon Athena Amazon CloudSearch Amazon QuickSight AWS Data Pipeline Amazon Machine Learning AWS IoT ଟछଟ༷ͳαʔϏε܈ σʔλҠߦ σʔλϕʔε ετϨʔδ ੳ ݕࡧ *P5 ίϯϐϡʔςΟϯά ػցֶश AWS Glue
ͲͷαʔϏεΛ͏͖͔ʁ Ͳ͏ͬͯ"84ʹσʔλΛΞοϓϩʔυ͢Δʁ σʔλͷՃͲ͜ͰΔʁ&$ʁ 3FETIJGUͬͨํ͕Α͍ʁ3%4Ͱେৎʁ
σʔλੳΞʔΩςΫνϟͷݪଇ wσʔλੳΛύΠϓϥΠϯͰߟ͑Δ w ऩूɺલॲཧɺอଘɺੳɺՄࢹԽ w"84ͷ֤αʔϏε͕Կʹ࠷దԽ͞Ε͍ͯΔ͔ʁ·ͨ ࠷దԽ͞Εͨ݁ՌͷτϨʔυΦϑԿ͔ʁΛཧղ͢Δ w εϧʔϓοτɺϨΠςϯγʔɺεέʔϥϏϦςΟɺ ߹ੑɺѻ͑ΔσʔλλΠϓʗσʔλߏɺ
ΧελϚΠζੑɺίετɺ্ݶ w֤αʔϏεͷಛΛཧղ্ͨ͠ͰɺύΠϓϥΠϯ্ͷ ͦΕͧΕͷδϣϒʹ࠷దͳαʔϏεΛબ͢Δ
ϏοάσʔλύΠϓϥΠϯ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ϏοάσʔλύΠϓϥΠϯอଘ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
σʔλ4ʹू Amazon S3 Amazon EC2 AWS Aurora Amazon Redshift AWS
Lambda Amazon ES Amazon EMR Amazon CloudSearch Amazon QuickSight Amazon Athena AWS IoT Amazon Kinesis Streams Amazon Kinesis Firehose AWS DMS AWS Batch AWS DynamoDB
ϏοάσʔλύΠϓϥΠϯऩू લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ऩू ϑΝΠϧ ετϦʔϜ w ച্σʔλ w ސ٬Ϛελ w Ϛελ w
ߦಈϩά w ΫϦοΫετϦʔϜ w *P5ηϯαʔσʔλ ɏ σʔλλΠϓʹண AWS IoT Amazon S3 Amazon Kinesis Streams Amazon Kinesis Firehose … … …
ϏοάσʔλύΠϓϥΠϯલॲཧ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
લॲཧ ϦΞϧλΠϜ όον w ࣌ؒຖɺ࣍ɺि࣍ɺ w εϧʔϓοτॏࢹ w ஞ࣍ॲཧ w
ϨΠςϯγʔॏࢹ Amazon EMR Amazon EC2 AWS Lambda Amazon Kinesis Analytics ɏ ϫʔΫϩʔυʹண ƽ
ϏοάσʔλύΠϓϥΠϯੳ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ੳ σʔλߏ ΞΫηεύλʔϯ σʔλετΞ ੳ ݻఆ͞ΕͨεΩʔϚ ෳࡶͳϦϨʔγϣφϧΫΤϦɺ ଟͷϢʔβʔ͔Βͷಉ࣌ΞΫηε 3%4 େنσʔλͷूܭɾநग़
3FETIJGU 4 3FETIJGU4QFDUSVN εΩʔϚϨε γϯϓϧͳΫΤϦɺ͍ϨΠςϯγʔ %ZOBNP%# ੜσʔλʹର͢ΔΞυϗοΫੳ 4 &.3 1SFTUP *NQBMB 4QBSL "UIFOB "UIFOB શจݕࡧ &MBTUJD4FBSDI4FSWJDF $MPVE4FBSDI ɏ σʔλߏɺΞΫηεύλʔϯʹண
ϏοάσʔλύΠϓϥΠϯՄࢹԽ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ՄࢹԽ ఏڙํ๏ ՄࢹԽπʔϧ ηϧϑ#* "NB[PO2VJDL4JHIU ఆܕϨϙʔτ ΧελϜ8FCΞϓϦέʔγϣϯ w %ZOBNP%# -BNCEB
"1*(BUFXBZ 4 w 3%4 &$ ϩάੳͷμογϡϘʔυ "NB[PO&MBTUJD4FBSDI4FSWJDF ,JCBOB ɏ σʔλͷఏڙํ๏ʹண͢Δ w "84αʔϏεͷબࢶগͳ͍αʔυύʔςΟͷ πʔϧݕ౼ ǟ ǟ
͜ͷηογϣϯʹ͍ͭͯ ύΠϓϥΠϯͰߟ͑Δσʔλੳج൫ɹ σʔλੳϦϑΝϨϯεΞʔΩςΫνϟ
ϦϑΝϨϯεΞʔΩςΫνϟ wǡ ച্ੳ wΦϯϓϨϛεͷ%#ʹ͋Δച্σʔλ wҎԼͷੳڥΛఏڙ wੜσʔλʹର͢ΔΞυϗοΫੳ wηϧϑ#* wఆܕϨϙʔτ
ച্ੳσʔλऩू ੜσʔλ σʔλ4ʹू σʔλΞοϓϩʔυ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳੜσʔλͷΞυϗοΫੳ Amazon Athena ੜσʔλ σʔλΞοϓϩʔυ ΞυϗοΫੳ "UIFOBͰ ΞυϗοΫੳ ǟ
Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳσʔλมɺܗ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon EMR σʔλΞοϓϩʔυ ΞυϗοΫੳ
σʔλมɺܗ &.3Ͱσʔλมɺܗ ؆қͳॲཧͰ͋Ε&$Ͱࡁ·ͤΔ͜ͱ ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳ3FETIJGUऔࠐΈ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ όοναʔόʔͰ $01:ίϚϯυΛ࣮ߦ ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳ3FETIJGUऔࠐΈ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ σΟϝϯγϣϯ ςʔϒϧ 3FETIJGU 4QFDUSVN ϑΝΫτςʔϒϧ4ʹɻ 4QFDUSVNͰΞΫηε ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳ#*πʔϧͰੳɺՄࢹԽ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon QuickSight Amazon Redshift
όοναʔόʔ Amazon EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ 4QFDUSVN 2VJDL4JHIUͰՄࢹԽɻ ͦͷଞ#*πʔϧ༻Մೳ ՄࢹԽ ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳϨϙʔτఏڙ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR 4QFDUSVN Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FEWJFX Ϩϙʔτදࣔ αʔόʔ ੳ୲ऀ (ਓʙेਓ) σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ ϨϙʔτӾཡऀ (100ਓʙ) Amazon QuickSight ǟ ՄࢹԽ Internet Internet σʔλϚʔτΛ3%4ʹอଘ͠ɺ ϨϙʔτӾཡऀʹ։์
ച্ੳશମ૾ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR 4QFDUSVN Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FEWJFX Ϩϙʔτදࣔ αʔόʔ ੳ୲ऀ (ਓʙेਓ) σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ ϨϙʔτӾཡऀ (100ਓʙ) Amazon QuickSight ǟ ՄࢹԽ Internet Internet
ϦϑΝϨϯεΞʔΩςΫνϟ w | 8FCථγεςϜ w8FC͔ΒථσʔλΛड͚͚ wථσʔλΛɺ wϦΞϧλΠϜʹूܭ wޙ#*πʔϧͰੳ
8FCථγεςϜථσʔλड ୲ऀ Amazon Kinesis Streams Internet ථσʔλड Internet "1*(BUFXBZΛϓϩΩγʹͯ͠
,JOFTJT4USFBNʹථσʔλΛొ Amazon API Gateway Amazon CloudFront ථϖʔδ4 $MPVE'SPOUͰ৴ Amazon S3 ථը໘
8FCථγεςϜ%#ͷొ ୲ऀ Amazon Kinesis Streams Amazon DynamoDB AWS Lambda
ථσʔλड ථσʔλొ Internet ථσʔλ -BNCEBͰ%ZOBNP%#ʹ ථσʔλΛొ Internet Amazon CloudFront ථσʔλड Amazon S3 Amazon API Gateway ථը໘
8FCථγεςϜϦΞϧλΠϜूܭ ୲ऀ Amazon Kinesis Streams Amazon Kinesis Analytics Amazon
Kinesis Streams AWS Lambda Amazon DynamoDB AWS Lambda ථσʔλड ථσʔλूܭ αϚϦσʔλొ ථσʔλొ Internet αϚϦσʔλ ථσʔλ ूܭͨ͠αϚϦσʔλΛ %ZOBNP%#ʹొ Internet Amazon CloudFront Amazon API Gateway Amazon S3 ථը໘ ,JOFTJT"OBMZUJDTͰ ථσʔλΛϦΞϧλΠϜूܭ
8FCථγεςϜΞʔΧΠϒ ୲ऀ Amazon Kinesis Streams Amazon DynamoDB AWS Lambda
ථσʔλड ථσʔλొ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ථσʔλ Internet Amazon CloudFront Amazon API Gateway Amazon S3 ථը໘ શථσʔλ,JOFTJT'JSFIPTFܦ༝Ͱ 4ʹΞʔΧΠϒ Amazon Kinesis Analytics Amazon Kinesis Streams ථσʔλूܭ AWS Lambda αϚϦσʔλొ
8FCථγεςϜσʔλࢀর ୲ऀ ౦ژϦʔδϣϯ Amazon Kinesis Streams Amazon DynamoDB AWS
Lambda ථσʔλड ථσʔλొ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ථσʔλ AWS Lambda %ZOBNP%# 4USFBN ථσʔλΛϦΞϧλΠϜʹࢀর Internet Amazon API Gateway Amazon CloudFront ूܭ݁Ռ4ʹΞοϓϩʔυ͠ $MPVE'SPOU͔Β৴ Amazon S3 ථը໘ Amazon Kinesis Analytics Amazon Kinesis Streams ථσʔλूܭ AWS Lambda αϚϦσʔλొ Amazon S3 ථཧը໘
8FCථγεςϜશମ૾ ୲ऀ ౦ژϦʔδϣϯ Amazon Kinesis Streams Amazon DynamoDB AWS
Lambda ථσʔλड ථσʔλొ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ථσʔλ AWS Lambda %ZOBNP%# 4USFBN Internet Amazon API Gateway Amazon CloudFront Amazon S3 ථཧը໘ Amazon S3 ථը໘ Amazon Kinesis Analytics Amazon Kinesis Streams ථσʔλूܭ AWS Lambda αϚϦσʔλొ
·ͱΊ wσʔλੳΛύΠϓϥΠϯͰߟ͑Δ w"84ͷ֤αʔϏε͕Կʹ࠷దԽ͞Ε͍ͯΔ͔ʁ·ͨ ࠷దԽ͞Εͨ݁ՌͷτϨʔυΦϑԿ͔ʁΛཧղ͢Δ w֤αʔϏεͷಛΛཧղ্ͨ͠ͰɺύΠϓϥΠϯ্ͷ ͦΕͧΕͷδϣϒʹ࠷దͳαʔϏεΛબ͢Δ
׀幠耮ָ֮הֲ׀ְׂת׃կ
None