Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
データサイエンスのためのAWSの使い方入門
Search
Show Murai
July 21, 2017
Technology
1
920
データサイエンスのためのAWSの使い方入門
データサイエンスのためのAWSの使い方入門
サポーターズCoLab勉強会 Jul 20 2017
村井 翔太朗
Show Murai
July 21, 2017
Tweet
Share
More Decks by Show Murai
See All by Show Murai
システムを構築するときに 気をつける10のこと
showmurai
4
140
監視入門!監視で困ってませんか?
showmurai
0
290
AWS vs GCP 今から作るならどちらがいいの 20180330
showmurai
1
1.3k
システムを構築するときに 気をつける10のこと
showmurai
0
1.5k
KubernetesでCLIを快適を使いたい
showmurai
0
1.8k
AWS vs GCP 今から作る ならどっちがいいの!?
showmurai
52
82k
Other Decks in Technology
See All in Technology
KubeConにproposalを送りたい人へのアドバイス
sat
PRO
3
270
web-application-security
matsuihidetoshi
1
190
よく聞くけど使ったことないソフトウェアNo.1 KafkaとSnowflake
foursue
4
510
TechFeed Experts Night#27 〜 フロントエンドフレームワーク最前線 (Svelte)
baseballyama
2
600
MLOpsの「壁」を乗り越える、LINEヤフーの Data Quality as Code
lycorptech_jp
PRO
8
630
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions (PSS)
pacificspatialsolutions
0
340
EMとして2023年度に頑張ったこと / What we did well in FY2023 as a EM
pauli
1
260
認知症フレンドリーテックとスタックチャン
naokiuc
0
280
ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation
lmi
1
150
生産性向上チームの紹介
cybozuinsideout
PRO
1
930
BPStudyの200回を中心にIT業界を振り返る。そしてこれから
haru860
3
410
M5stackで使用できるpHセンサの開発
shinrinakamura
0
200
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
20
1.6k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
221
21k
Thoughts on Productivity
jonyablonski
60
3.9k
Documentation Writing (for coders)
carmenintech
60
4k
Optimising Largest Contentful Paint
csswizardry
12
2.4k
Statistics for Hackers
jakevdp
790
220k
Rails Girls Zürich Keynote
gr2m
91
13k
From Idea to $5000 a Month in 5 Months
shpigford
378
45k
Ruby is Unlike a Banana
tanoku
96
10k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
8
1.3k
Web development in the modern age
philhawksworth
203
10k
Transcript
σʔλαΠΤϯεͷͨΊͷ AWSͷ͍ํೖ αϙʔλʔζCoLabษڧձ Jul 20 2017 ଜҪ ᠳଠ࿕
ࣗݾհ • ଜҪᠳଠ࿕ • גࣜձࣾαΠόʔΤʔδΣϯτ • ΞυςΫຊ෦ ΠϯϑϥνʔϜ • ΠϯϑϥΤϯδχΞ
※ຊͷൃදݸਓͷݟղͰ͋Γɺॴଐ͢Δ৫ͷެࣜݟղͰ͋Γ·ͤΜ
͓·͑AWSৄ͍͠ͷʁ
ਓฒΈʹͬͯΔ(ͣ) • AWSྺɿ3͘Β͍ • AWSͰ͖ͳαʔϏεɿRedshiftɺLambda • AWSೝఆɿAWS SAA • Re:Invent
2015 ࢀՃ͠·ͨ͠
ΊͪΌͪ͘Όৄ͋͘͠Γ·ͤΜ ઌAWS SAPམͪ·ͨ͠…
օ༷ʹ࣭͍ͤͯͩ͘͞͞
σʔλαΠΤϯςΟετͷํ
σʔλੳ͠ͳ͍͚Ͳ σʔλੳج൫ͷߏஙΛ͢Δํ
ͳΜͰͦͦࠓͷ͠ σʔλαΠΤϯεͱAWSͳͷʁ
༑ਓ ࠷ۙɺσʔλαΠΤϯεςΟετʹͳͬͨΘ ͦΕͦΕɺ͓څྉ͗ΐ͏͞Μ͏ͯΔΜΖ ଜҪ ϕϯνϟʔ͔Β·ͩ·ͩ͜Ε͔Βɻ ·ͩGPUͱ͔αʔόʔΛങ͏͓ۚͳ͍͠ɻ Ͱֶशͤ͞Δͷʹ͔͔ΔͷΑͶʔ ͋ΕɺΫϥυͬͯͳ͍ͷʁ Ϋϥυͬͯͬͨ͜ͱͳ͍͚Ͳߴ͍Ͱ͠ΐ ͋ͱ͏ͷͦ͠͏͠
ωοτϫʔΫͱ͔ߏஙͱ͔Ͱ͖ͳ͍͠ ͦΜͳ͜ͱ͍͜͠ͱ͋ΒΜͰ ͦ͏ͳͷʁ ΄ͳࠓίʔώʔ͓͝Δ͔Β͓͑ͯ͠ (130ԁͰͪΐͬͱ…)
ಉ྅ ػցֶश༻ͷαʔόʔͱ σʔλʹੳ༻ͷΫΤϦΛ͛ΔDBͱ ଞ෦ॺͱσʔλͷڞ༗͍ͨ͠ ͬͯݴΘΕͨͷ͚Ͳ૬ஊͤͯ͞ʔ ͓ɺ·͍Ͳʂ ͜ΕͨͿΜΫϥυҊ݅Ͱ͢Θ ଜҪ ͑ɺࣗࣾαʔόʔ͡Όͳ͘Ϋϥυͳͷʁ نʹΑΓ·͕͢Ϋϥυͷ΄͏͕
։ൃεϐʔυ͕͘ɺίετ҆͘ɺ ӡ༻ָ͕Ͱ͢ͶΜ ΄͏΄͏ɺ΄ͳΘ͔ͬͯΔΜͬͨΒ ͜ͷ͓ئ͍͍͍ͯ͠ʁ (͑…ԶʹλεΫ͕) ͜Ε͚ͩͱΕ·Μɻ ҰճͪΌΜͱώΞϦϯάͤͯ͞Β͍·͢Θɻ
ͦΜͳਓੈͷதʹ ͪΐͬͱ͚͍ͩΔͷͰͳ͍͔ͱɻ
ຊͷରऀ • Ϋϥυ͍͍ͨσʔλαΠΤϯςΟετͷํ • σʔλαΠΤϯςΟετ(Ά͍ਓ)ʹ ج൫ͷߏஙΛ͓ئ͍͞Εͨํ • AWSͬͨ͜ͱͳ͍ํ • ࣗࣾ/ֶߍͰσʔλੳج൫Λ࣋ͬͯͳ͍ํ
ࠓ͓͢Δ͜ͱ • AWSͱ • σʔλαΠΤϯεͱ • AWSαʔϏεհ • Amazon EC2
• Amazon EMR • Amazon S3 • Amazon Redshift • ΫϥυͰࣄނΛ͙ͨΊʹ
AWSͱʁ
AWS(Amazon Web Service) ͱ ΫϥυίϯϐϡʔςΟϯάαʔϏε →Ϩϯλϧαʔόʔͷ͍ͭ͢͝ ʮΠϯϑϥ͕WebαʔϏεʹͳͬͨʯ AWS ʹ͍ͭͯ https://aws.amazon.com/jp/about-aws/
AWSੈքதʹ͋Δ • Ϧʔδϣϯ • େͳͲཧతʹΕͨྖҬ • 14ͭͷϦʔδϣϯ • ౦ژɺόʔδχΞɺϩϯυϯ •
ΞϕΠϥϏϦςΟʔκʔϯ • 1 ͭͷϦʔδϣϯʹෳͷͦΕͧΕಠཱͨ͠ϩέʔγϣϯ • ྫ. ౦ژϦʔδϣϯͷAZɿɺཱ ※࣮ࡍʹެ։͞Ε͍ͯ·ͤΜ EC2 ϦʔδϣϯͱΞϕΠϥϏϦςΟʔκʔϯ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
AWSͰग़དྷΔ͜ͱ(΄ΜͷҰ෦) • ؆୯ʹߏஙͰ͖Δίϯϐϡʔτ(αʔόʔ) • EC2 • ΫϦοΫ͚ͩͰग़དྷΔσʔλϕʔεͷߏங • RDS •
੍ݶແ͠ͷσʔλετϨʔδ • S3 • ετϦʔϜॲཧ • Kinesis (Stream)
σʔλαΠΤϯεͱʁ σʔλΛऩूɺੳɺར༻ͯ͠༗ޮ׆༻͢Δٕज़·ֶͨ
σʔλαΠΤϯςΟετͱ σʔλαΠΤϯεྗɺ σʔλΤϯδχΞϦϯάྗΛϕʔεʹ σʔλ͔ΒՁΛग़͠ɺ Ϗδωε՝ʹ͑Λग़͢ ϓϩϑΣογϣφϧ ※͜͜ͰʮϏδωεʯͱࣾձʹʹཱͭҙຯͷ͋Δ׆ಈશൠΛࢦ͢ Ұൠࣾஂ๏ਓσʔλαΠΤϯςΟετڠձ σʔλαΠΤϯςΟετͷϛογϣϯɺεΩϧηοτɺఆٛɺεΩϧϨϕϧΛൃද http://www.datascientist.or.jp/news/2014/pdf/1210.pdf
ͳͥσʔλαΠΤϯεͰ AWS(Ϋϥυ)Λ͏ͷ͔
σʔλੳʹٻΊΒΕΔ͜ͱ • σʔλΞφϦςΟΫε • େ͖͍σʔλΛ҆Ձʹѻ͍͍ͨ • σʔλΣΞϋεʹੳΫΤϦΛ࣮ߦ͍ͨ͠ • ؆୯ʹՄࢹԽ͍ͨ͠ •
ػցֶश࣮ߦج൫ • ֶश࣌”͚ͩ”ඞཁʹͳΔେྔͷGPU • ӡ༻ΛͳΔͨ͘͘͠ͳ͍ • ߏͳͲࣗ༝͕ߴ͍
ඞཁͳ࣌ʹ ඞཁͳϦιʔε͚ͩΛ֬อ͠ ඞཁͳ͚ͩੳ͍ͨ͠ Ϋϥυ͕࠷దͰ͢Αʂ
σʔλੳ͢Δࡍʹ AWSͷͲͷαʔϏε͕͑Δ͔
հ͢ΔαʔϏε Amazon EC2 Amazon S3 Amazon EMR Amazon Redshift ԾαʔόߏஙαʔϏε
εέʔϥϒϧͰ ͚ͬͨͩͷैྔ՝ۚ ϏοάσʔλϑϨʔϜϫʔΫΛ ؆୯ʹߏஙͰ͖ΔϚωʔδυαʔϏε HadoopɺSparkΫϥελʔ͕؆୯ʹߏஙͰ͖Δ σʔλετϨʔδαʔϏε ແ੍ݶʹσʔλ͕อͰ͖Δ ੩తίϯςϯπͷWebϗεςΟϯάՄೳ AWS͕ఏڙ͢Δ શϚωʔδυͷσʔλΣΞϋε SQL͕࣮ߦͰ͖ϖλόΠτنͷ σʔλΛѻ͑Δ
Amazon EC2
Amazon EC2ͱ • ΫϥυίϯϐϡʔςΟϯά • ͍ΘΏΔԾαʔόͷߏங • ྉۚମܥ • ίϯϐϡʔτͷىಈ࣌ؒʹԠͯ͡՝ۚ
• ϘϦϡʔϜʹର͢Δ՝ۚ • ωοτϫʔΫసૹྔʹର͢Δ՝ۚ • ๛ͳΠϯελϯελΠϓ Amazon EC2 ͱ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/concepts.html
Amazon EC2ͷҙ • ॳظόʔδχΞ෦ϦʔδϣϯͰىಈ • ηΩϡϦςΟάϧʔϓͰϙʔτΛղ์͠ͳ͍ͱ ωοτϫʔΫͷૄ௨͕ग़དྷͳ͍ • ηΩϡϦςΟάϧʔϓͰશެ։[0.0.0.0/0]Λ ઃఆ͢ΔࡍҙΛʂ
• ύϒϦοΫIP/ElasticIPΛ༩͢Δඞཁ͋Γ (VPNͳͲଓͯ͠Δ߹Λআ͘) • ՝ۚ1࣌ؒ୯Ґ • 1ͬͯɺ59ͬͯಉ͡ྉۚ
EC2εϙοτΠϯελϯε • Amazonʹམͱ͞ΕΔ͔͠Εͳ͍͚ͲɺՁ֨ͳΠϯελϯε • ೖࡳՁ֨ > ࢢՁ֨ͳΒࢢՁ֨ͰΠϯελϯεΛىಈͰ͖Δ • ࢢՁ֨ >
ೖࡳՁ֨ͱͳΔͱΠϯελϯεऴྃ • 70%off͘Β͍Ͱ͑Δ(ݸਓ࣮) ※ΦϯσϚϯυΠϯελϯεͱͷൺֱ • ೖࡳΞυόΠβʔΛ͏ https://aws.amazon.com/jp/ec2/spot/bid-advisor/ Amazon EC2 εϙοτΠϯελϯε http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-spot-instances.html
ͱ͜ΖͰػցֶश͍ͨ͠ͷͰ GPUΛ͍͍ͨ
GPUͬͯͳʹʁ • Graphics Processing Unit • ݩʑɺը૾ॲཧͳͲ͚ʹ։ൃ͞Εͨ • ฒྻԋࢉ͕ಘҙͳϓϩηοα •
ߦྻԋࢉ͕ಘҙ ˠ σΟʔϓϥʔχϯάʹద͍ͯ͠Δ (16 $16
GPU͏ͱͳ͍ͥͷ͔ • ԋࢉग़དྷΔίΞ͕2000-4000ίΞ • ฒྻܭࢉॲཧʹ༗ར • CPU(Intel Xeonͱ͔)ͩͱ࠷େ48ίΞ͘Β͍ • ϝϞϦ͕ಠཱ͔ͭߴಡΈࠐΈ
• CPUͷϝϞϦόϯυ෯ΑΓ10ഒ͘Β͍͍ • σΟʔϓϥʔχϯάͰར༻͢Δߦྻܭࢉ ΛฒྻͰߦ͑ΔͷͰAIج൫Ͱར༻͕ਐΉ
ৄ͘͠GPUͷ͍ͭͯΓ͍ͨํ ͜ͷຊ͕͓͢͢ΊͰ͢ GPUΛࢧ͑Δٕज़ ஶऀɿHisa Ando ൃചɿ20176݄30 URLɿhttps://www.amazon.co.jp/dp/477419056X ※ΞϑΟϦϯΫͰ͋Γ·ͤΜ
AWSͰ͑ΔGPU
GPUΠϯελϯελΠϓ p2Πϯελϯε g2 Πϯελϯε g3Πϯελϯε ←new
ࡌGPUΠϯελϯε Πϯελϯε $16 W$16 ϝϞϦ (16 (16ϝϞϦ උߟ QYMBSHF
(# /7*%*"5FTMB, (# QYMBSHF (# (# QYMBSHF (# (# HYMBSHF (# /7*%*"(3*%, (# HYMBSHF (# (# HYMBSHF (# /7*%*"5FTMB. (# ౦ژϦʔδϣϯ Ͱ͑ͳ͍ HYMBSHF (# (# HYMBSHF (# (# ※vCPUεϨου
Amazon EMR
Amazon EMRͱ • EC2Πϯελϯε্ʹࢄॲཧج൫Λ ΫϥελʔͰߏங͢ΔαʔϏε • HadoopɺApache SparkڥͳͲ
Amazon EMRͷҙͳͲ • S3ʹ͋ΔΞϓϦέʔγϣϯΛࢦఆ࣮ͯ͠ߦͰ ͖ΔͷͰྑ͍ • ϩάS3ʹग़ྗͰ͖ΔͷͰϩά͕อଘग़དྷΔ • ࢹ͕͍͠ •
ΞϓϦέʔγϣϯଆͰৄࡉͳΤϥʔΛग़ྗͨ͠΄͏͕͍͍ • EMRͰޭ/ࣦഊ͔͠ग़ྗ͞Εͳ͍
Amazon S3
σʔλϨΠΫ • σʔλϨΠΫͱ • ੜϩάɺը૾ɾө૾ɺԻɺͳͲͷ ʮඇߏԽσʔλʯΛཧ͢ΔྖҬ • ඞཁͳ࣌ʹσʔλϨΠΫ͔ΒऔΓग़͠ɺ ͙͢͞·σʔλΛ׆༻Ͱ͖Δ GE
Announces First Data Lake Approach for Industrial Internet to Better Access, Analyze and Store Industrial-Strength Big Data http://www.businesswire.com/news/home/20140810005024/en/GE-Announces-Data-Lake-Approach-Industrial-Internet
Amazon S3 • AWSͷετϨʔδαʔϏε σʔλϨΠΫͱͯ͑͠Δ • ಛ • ٱੑ 99.999999999%
• 1ສݸͷΦϒδΣΫτͷ͏ͪ1ͭͷফࣦʹ1000ສ͔͔Δ • Մ༻ੑ 99.99% • S3αʔϏεμϯλΠϜ 5234ඵ • ࡉ͔͘ઃఆͰ͖ΔΞΫηείϯτϩʔϧ • ACLɺόέοτϙϦγʔɺAIMϙϦγʔ S3ͷΞΫηείϯτϩʔϧ·ͱΊ http://qiita.com/ryo0301/items/791c0a666feeea0a704c
Amazon S3ͷछྨ • S3 ඪ४ • ී௨ͷετϨʔδ • S3 ඪ४
– ԽετϨʔδ • ॏཁੑͷ͍࠶ੜՄೳͳσʔλ͚ • S3 ඪ४ – සΞΫηε • ΞΫηε͕গͳ͍σʔλ͚ • ΞʔΧΠϒ(Amazon Glacier) • ΊͬͨʹΞΫηε͞Εͳ͍σʔλ༻ https://aws.amazon.com/jp/s3/storage-classes/
Amazon S3ͷྉۚମܥ • ετϨʔδྉۚ • ͲΕ͘Β͍σʔλྔΛஔ͍ͯΔ͔ • σʔλసૹྔ • ͲΕ͘Β͍σʔλΛ࣋ͪग़͔ͨ͠
• APIίʔϧྉۚ • Ξοϓϩʔυ/আ/μϯϩʔυͳͲͷૢ࡞
Amazon S3ͷҙ • σʔλసૹྔ͕݁ߏߴ͍Ͱ͢ ex.1TB(1024GB)ΛΠϯλʔωοτܦ༝Ͱ μϯϩʔυͨ͠Β1.5ສԁ($143.36) • σϑΥϧτͷอઌ͕ ถࠃ౦෦(όʔδχΞ)
Amazon Redshift
Amazon Redshiftͱ • AWS͕ఏڙ͢ΔશϚωʔδυܕͷ σʔλΣΞϋε(DHW) • ϖλόΠτΫϥεʹεέʔϧ͢Δ • ྻࢦσʔλϕʔε •
ྉۚମܥΦϯσϚϯυྉۚ
Amazon Redshiftͷҙ • σʔλߏͷઃܭΛ͢Δ(͋ͨΓલ?) • ಛʹࢄΩʔɺιʔτΩʔɺ࣌ܥྻςʔϒϧׂ • ࣌ʑɺڧ੍ϝϯςφϯε͕ൃੜ͢Δ • ϦΞϧλΠϜॲཧʹෆ͖
• ͋͘·Ͱੳ༻ʹ͏ • ຊ൪ڥͰར༻͢Δ߹μϯλΠϜΛఆ͢Δ • μογϡϘʔυ༏ल
հ͔ͨͬͨ͠αʔϏε Amazon RDS AWS͕ఏڙ͢ΔϚωʔδυܕ ϦϨʔγϣφϧσʔλϕʔε Amazon Athena Amazon S3 ͷσʔλʹ
SQL Λ࣮ߦͰ͖ΔαʔϏε 6݄ʹ౦ژϦʔδϣϯ։࢝ Amazon Kinesis Amazon Elasticsearch AWS Lambda Amazon VPC Amazon CloudWatch Amazonͷఏڙ͢Δ ԾωοτϫʔΫαʔϏε جຊແྉͰར༻Մೳ ετϦʔϛϯάॲཧ͕ ग़དྷΔϚωʔδυαʔϏε αʔόʔϨεͰϓϩάϥϜΛ ࣮ߦͰ͖ΔαʔϏε ElasticsearchΛΤϯδϯ ͱͨ͠ϚωʔδυαʔϏε AWSͷࢹπʔϧ
ΫϥυͰࣄނΛ͙ͨΊʹ
ઈରΔ͖͜ͱ • ྉۚΞϥʔϜΛઃఆ͢Δ Ξϥʔτͱ௨ͰٻֹΛϞχλϦϯά http://docs.aws.amazon.com/ja_jp/awsaccountbilling/latest/aboutv2/monitor-charges.html • APIΩʔͷཧެ։ઈର͠ͳ͍ • Access key/Secret
Keyඞͣެ։͠ͳ͍Α͏ʹ • ʮॳ৺ऀ͕AWSͰϛεͬͯෆਖ਼ར༻͞Εͯ$6,000ٻɺٽ͖ͦ͏ʹͳ͓ͬͨʯͳͲ • ݟੵΓඞ͓ͣ͜ͳ͏ • AWS ؆қݟੵΓπʔϧ http://calculator.s3.amazonaws.com/index.html?lng=ja_JP • AWS ՝ۚମܥͱݟੵΓํ๏ʹ͍ͭͯ https://aws.amazon.com/jp/how-to-understand-pricing/ • ηΩϡϦςΟάϧʔϓ[ssh 0.0.0.0/0]ͱ͔ઃఆ͠ͳ͍ • ݁ߏϒϧʔτϑΥʔε(૯Γ߈ܸ)ͱ͔ɺ੬ऑੑ߈ܸ͕͋Γ·͢
࣭λΠϜ
AWSσʔλαΠΤϯε Γ͍ͨͻͱ ϚδͰઈࢍืूத ※ΦϑΟεݟֶ͚ͩͰେৎͰ͢ αΠόʔΤʔδΣϯτ ΞυςΫελδΦ https://adtech.cyberagent.io/
͋Γ͕ͱ͏͍͟͝·ͨ͠
ࢀߟใ AWS υΩϡϝϯτ https://aws.amazon.com/jp/documentation/ AWS ΫϥυαʔϏε׆༻ࢿྉू https://aws.amazon.com/jp/aws-jp-introduction/ Youtube AWSެࣜνϟϯωϧ(Re:InventͷηογϣϯͳͲ) ӳޠ
https://www.youtube.com/user/AmazonWebServices ຊޠ https://www.youtube.com/user/AmazonWebServicesJP AWSΫϥυσβΠϯύλʔϯ http://aws.clouddesignpattern.org/index.phpύλʔϯ Developers.IO http://dev.classmethod.jp/ DeNA TechCon 2017 ͱ Developers Summit 2017 ͰDeNAͷػցֶशج൫ͱੳج൫ͷߨԋΛ͠·ͨ͠ http://blog.livedoor.jp/sonots/archives/49502478.html