Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AWSで始めるビッグデータ分析 Developers.IO 2017 WORLD in 名古屋
Search
Yutaka Yawata
September 28, 2017
Technology
1
160
AWSで始めるビッグデータ分析 Developers.IO 2017 WORLD in 名古屋
Developers.IO 2017 WORLD in 名古屋 登壇資料
Yutaka Yawata
September 28, 2017
Tweet
Share
More Decks by Yutaka Yawata
See All by Yutaka Yawata
AWSで始めるビッグデータ分析 Developers.IO 2017
yuyawata
1
3.1k
AWSで作るオムニチャネル戦略実現のためのマーケティング・プラットフォーム
yuyawata
1
2k
Other Decks in Technology
See All in Technology
AIを使っていい感じにE2Eテストを書けるようになるまで / Trying to Write Good E2E Tests with AI
katawara
3
1.8k
[TechNight #91] Oracle Database 最新パフォーマンス分析手法
oracle4engineer
PRO
2
110
経理出身PdMがAIプロダクト開発を_ハンズオンで学んだ話.pdf
shunsukenarita
1
200
自分がLinc’wellで提供しているプロダクトを理解するためにやったこと
murabayashi
1
160
Kiro Hookを Terraformで検証
ao_inoue
0
130
FAST導入1年間のふりかえり〜現実を直視し、さらなる進化を求めて〜 / Review of the first year of FAST implementation
wooootack
1
170
Wasmで社内ツールを作って配布しよう
askua
0
150
Webの技術とガジェットで那須の子ども達にワクワクを! / IoTLT_20250720
you
PRO
0
130
Expertise as a Service via MCP
yodakeisuke
1
150
2025-07-25 NOT A HOTEL TECH TALK ━ スマートホーム開発の最前線 ━ SOFTWARE
wakinchan
0
160
AI駆動開発 with MixLeap Study【大阪支部 #3】
lycorptech_jp
PRO
0
260
会社もクラウドも違うけど 通じたコスト削減テクニック/Cost optimization strategies effective regardless of company or cloud provider
aeonpeople
2
320
Featured
See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.8k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Balancing Empowerment & Direction
lara
1
500
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.9k
Visualization
eitanlees
146
16k
Unsuck your backbone
ammeep
671
58k
Building an army of robots
kneath
306
45k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Thoughts on Productivity
jonyablonski
69
4.8k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
850
Transcript
"84Ͱ࢝ΊΔϏοάσʔλੳ %FWFMPQFST*0803-%JO໊ݹ ݄ Ϋϥεϝιουגࣜձࣾീ഼๛ DNEFWJP
ࣗݾհ wീ഼๛ʢ:VUBLB:BXBUBʣ wϓϦηʔϧε୲ wݩ&31ύοέʔδͷج൫ίϯαϧ ࣗݾհ
͜ͷηογϣϯʹ͍ͭͯ ύΠϓϥΠϯͰߟ͑Δσʔλੳج൫ɹ σʔλੳϦϑΝϨϯεΞʔΩςΫνϟ
ηογϣϯͷରऀ w"84ͰσʔλੳΛ࢝Ί͍ͨ w"84Ͱσʔλੳج൫Λߏங͢ΔͨΊͷ ϕετϓϥΫςΟεΛΓ͍ͨ w"84ͷར༻ܦݧ͕͋Δ 71$ &$ 3%4 4 FUD
͜ͷηογϣϯͰ͓͠͠ͳ͍͜ͱ wͳͥ"84͔ʁ w֤"84αʔϏεͷৄࡉͳػೳεϖοΫ wػցֶशɺ"*
͜ͷηογϣϯʹ͍ͭͯ ύΠϓϥΠϯͰߟ͑Δσʔλੳج൫ɹ σʔλੳϦϑΝϨϯεΞʔΩςΫνϟ
༷ʑͳϏοάσʔλ ɨ 104σʔλ ϞόΠϧΞϓϦ ߦಈϩά ΫϦοΫ ετϦʔϜ 4/4
ηϯλʔσʔλ ɓ
σʔλੳͰར༻Ͱ͖Δ"84αʔϏε Amazon DynamoDB Amazon RDS Amazon Redshift Amazon S3
AWS Snowball AWS Lambda AWS Batch Amazon EC2 AWS DMS Amazon ES Amazon EMR Amazon Kinesis Amazon Athena Amazon CloudSearch Amazon QuickSight AWS Data Pipeline Amazon Machine Learning AWS IoT ଟछଟ༷ͳαʔϏε܈ σʔλҠߦ σʔλϕʔε ετϨʔδ ੳ ݕࡧ *P5 ίϯϐϡʔςΟϯά ػցֶश AWS Glue
ͲͷαʔϏεΛ͏͖͔ʁ Ͳ͏ͬͯ"84ʹσʔλΛΞοϓϩʔυ͢Δʁ σʔλͷՃͲ͜ͰΔʁ&$ʁ 3FETIJGUͬͨํ͕Α͍ʁ3%4Ͱେৎʁ
σʔλੳΞʔΩςΫνϟͷݪଇ wσʔλੳΛύΠϓϥΠϯͰߟ͑Δ w ऩूɺલॲཧɺอଘɺੳɺՄࢹԽ w"84ͷ֤αʔϏε͕Կʹ࠷దԽ͞Ε͍ͯΔ͔ʁ·ͨ ࠷దԽ͞Εͨ݁ՌͷτϨʔυΦϑԿ͔ʁΛཧղ͢Δ w εϧʔϓοτɺϨΠςϯγʔɺεέʔϥϏϦςΟɺ ߹ੑɺѻ͑ΔσʔλλΠϓʗσʔλߏɺ
ΧελϚΠζੑɺίετɺ্ݶ w֤αʔϏεͷಛΛཧղ্ͨ͠ͰɺύΠϓϥΠϯ্ͷ ͦΕͧΕͷδϣϒʹ࠷దͳαʔϏεΛબ͢Δ
ϏοάσʔλύΠϓϥΠϯ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ϏοάσʔλύΠϓϥΠϯอଘ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
σʔλ4ʹू Amazon S3 Amazon EC2 AWS Aurora Amazon Redshift AWS
Lambda Amazon ES Amazon EMR Amazon CloudSearch Amazon QuickSight Amazon Athena AWS IoT Amazon Kinesis Streams Amazon Kinesis Firehose AWS DMS AWS Batch AWS DynamoDB
ϏοάσʔλύΠϓϥΠϯऩू લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ऩू ϑΝΠϧ ετϦʔϜ w ച্σʔλ w ސ٬Ϛελ w Ϛελ w
ߦಈϩά w ΫϦοΫετϦʔϜ w *P5ηϯαʔσʔλ ɏ σʔλλΠϓʹண AWS IoT Amazon S3 Amazon Kinesis Streams Amazon Kinesis Firehose … … …
ϏοάσʔλύΠϓϥΠϯલॲཧ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
લॲཧ ϦΞϧλΠϜ όον w ࣌ؒຖɺ࣍ɺि࣍ɺ w εϧʔϓοτॏࢹ w ஞ࣍ॲཧ w
ϨΠςϯγʔॏࢹ Amazon EMR Amazon EC2 AWS Lambda Amazon Kinesis Analytics ɏ ϫʔΫϩʔυʹண ƽ
ϏοάσʔλύΠϓϥΠϯੳ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ੳ σʔλߏ ΞΫηεύλʔϯ σʔλετΞ ੳ ݻఆ͞ΕͨεΩʔϚ ෳࡶͳϦϨʔγϣφϧΫΤϦɺ ଟͷϢʔβʔ͔Βͷಉ࣌ΞΫηε 3%4 େنσʔλͷूܭɾநग़
3FETIJGU 4 3FETIJGU4QFDUSVN εΩʔϚϨε γϯϓϧͳΫΤϦɺ͍ϨΠςϯγʔ %ZOBNP%# ੜσʔλʹର͢ΔΞυϗοΫੳ 4 &.3 1SFTUP *NQBMB 4QBSL "UIFOB "UIFOB શจݕࡧ &MBTUJD4FBSDI4FSWJDF $MPVE4FBSDI ɏ σʔλߏɺΞΫηεύλʔϯʹண
ϏοάσʔλύΠϓϥΠϯՄࢹԽ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ՄࢹԽ ఏڙํ๏ ՄࢹԽπʔϧ ηϧϑ#* "NB[PO2VJDL4JHIU ఆܕϨϙʔτ ΧελϜ8FCΞϓϦέʔγϣϯ w %ZOBNP%# -BNCEB
"1*(BUFXBZ 4 w 3%4 &$ ϩάੳͷμογϡϘʔυ "NB[PO&MBTUJD4FBSDI4FSWJDF ,JCBOB ɏ σʔλͷఏڙํ๏ʹண͢Δ w "84αʔϏεͷબࢶগͳ͍αʔυύʔςΟͷ πʔϧݕ౼ ǟ ǟ
͜ͷηογϣϯʹ͍ͭͯ ύΠϓϥΠϯͰߟ͑Δσʔλੳج൫ɹ σʔλੳϦϑΝϨϯεΞʔΩςΫνϟ
ϦϑΝϨϯεΞʔΩςΫνϟ wǡ ച্ੳ wΦϯϓϨϛεͷ%#ʹ͋Δച্σʔλ wҎԼͷੳڥΛఏڙ wੜσʔλʹର͢ΔΞυϗοΫੳ wηϧϑ#* wఆܕϨϙʔτ
ച্ੳσʔλऩू ੜσʔλ σʔλ4ʹू σʔλΞοϓϩʔυ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳੜσʔλͷΞυϗοΫੳ Amazon Athena ੜσʔλ σʔλΞοϓϩʔυ ΞυϗοΫੳ "UIFOBͰ ΞυϗοΫੳ ǟ
Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳσʔλมɺܗ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon EMR σʔλΞοϓϩʔυ ΞυϗοΫੳ
σʔλมɺܗ &.3Ͱσʔλมɺܗ ؆қͳॲཧͰ͋Ε&$Ͱࡁ·ͤΔ͜ͱ ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳ3FETIJGUऔࠐΈ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ όοναʔόʔͰ $01:ίϚϯυΛ࣮ߦ ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳ3FETIJGUऔࠐΈ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ σΟϝϯγϣϯ ςʔϒϧ 3FETIJGU 4QFDUSVN ϑΝΫτςʔϒϧ4ʹɻ 4QFDUSVNͰΞΫηε ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳ#*πʔϧͰੳɺՄࢹԽ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon QuickSight Amazon Redshift
όοναʔόʔ Amazon EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ 4QFDUSVN 2VJDL4JHIUͰՄࢹԽɻ ͦͷଞ#*πʔϧ༻Մೳ ՄࢹԽ ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳϨϙʔτఏڙ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR 4QFDUSVN Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FEWJFX Ϩϙʔτදࣔ αʔόʔ ੳ୲ऀ (ਓʙेਓ) σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ ϨϙʔτӾཡऀ (100ਓʙ) Amazon QuickSight ǟ ՄࢹԽ Internet Internet σʔλϚʔτΛ3%4ʹอଘ͠ɺ ϨϙʔτӾཡऀʹ։์
ച্ੳશମ૾ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR 4QFDUSVN Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FEWJFX Ϩϙʔτදࣔ αʔόʔ ੳ୲ऀ (ਓʙेਓ) σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ ϨϙʔτӾཡऀ (100ਓʙ) Amazon QuickSight ǟ ՄࢹԽ Internet Internet
ϦϑΝϨϯεΞʔΩςΫνϟ w | 8FCථγεςϜ w8FC͔ΒථσʔλΛड͚͚ wථσʔλΛɺ wϦΞϧλΠϜʹूܭ wޙ#*πʔϧͰੳ
8FCථγεςϜථσʔλड ୲ऀ Amazon Kinesis Streams Internet ථσʔλड Internet "1*(BUFXBZΛϓϩΩγʹͯ͠
,JOFTJT4USFBNʹථσʔλΛొ Amazon API Gateway Amazon CloudFront ථϖʔδ4 $MPVE'SPOUͰ৴ Amazon S3 ථը໘
8FCථγεςϜ%#ͷొ ୲ऀ Amazon Kinesis Streams Amazon DynamoDB AWS Lambda
ථσʔλड ථσʔλొ Internet ථσʔλ -BNCEBͰ%ZOBNP%#ʹ ථσʔλΛొ Internet Amazon CloudFront ථσʔλड Amazon S3 Amazon API Gateway ථը໘
8FCථγεςϜϦΞϧλΠϜूܭ ୲ऀ Amazon Kinesis Streams Amazon Kinesis Analytics Amazon
Kinesis Streams AWS Lambda Amazon DynamoDB AWS Lambda ථσʔλड ථσʔλूܭ αϚϦσʔλొ ථσʔλొ Internet αϚϦσʔλ ථσʔλ ूܭͨ͠αϚϦσʔλΛ %ZOBNP%#ʹొ Internet Amazon CloudFront Amazon API Gateway Amazon S3 ථը໘ ,JOFTJT"OBMZUJDTͰ ථσʔλΛϦΞϧλΠϜूܭ
8FCථγεςϜΞʔΧΠϒ ୲ऀ Amazon Kinesis Streams Amazon DynamoDB AWS Lambda
ථσʔλड ථσʔλొ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ථσʔλ Internet Amazon CloudFront Amazon API Gateway Amazon S3 ථը໘ શථσʔλ,JOFTJT'JSFIPTFܦ༝Ͱ 4ʹΞʔΧΠϒ Amazon Kinesis Analytics Amazon Kinesis Streams ථσʔλूܭ AWS Lambda αϚϦσʔλొ
8FCථγεςϜσʔλࢀর ୲ऀ ౦ژϦʔδϣϯ Amazon Kinesis Streams Amazon DynamoDB AWS
Lambda ථσʔλड ථσʔλొ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ථσʔλ AWS Lambda %ZOBNP%# 4USFBN ථσʔλΛϦΞϧλΠϜʹࢀর Internet Amazon API Gateway Amazon CloudFront ूܭ݁Ռ4ʹΞοϓϩʔυ͠ $MPVE'SPOU͔Β৴ Amazon S3 ථը໘ Amazon Kinesis Analytics Amazon Kinesis Streams ථσʔλूܭ AWS Lambda αϚϦσʔλొ Amazon S3 ථཧը໘
8FCථγεςϜશମ૾ ୲ऀ ౦ژϦʔδϣϯ Amazon Kinesis Streams Amazon DynamoDB AWS
Lambda ථσʔλड ථσʔλొ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ථσʔλ AWS Lambda %ZOBNP%# 4USFBN Internet Amazon API Gateway Amazon CloudFront Amazon S3 ථཧը໘ Amazon S3 ථը໘ Amazon Kinesis Analytics Amazon Kinesis Streams ථσʔλूܭ AWS Lambda αϚϦσʔλొ
·ͱΊ wσʔλੳΛύΠϓϥΠϯͰߟ͑Δ w"84ͷ֤αʔϏε͕Կʹ࠷దԽ͞Ε͍ͯΔ͔ʁ·ͨ ࠷దԽ͞Εͨ݁ՌͷτϨʔυΦϑԿ͔ʁΛཧղ͢Δ w֤αʔϏεͷಛΛཧղ্ͨ͠ͰɺύΠϓϥΠϯ্ͷ ͦΕͧΕͷδϣϒʹ࠷దͳαʔϏεΛબ͢Δ
׀幠耮ָ֮הֲ׀ְׂת׃կ
None