データサイエンスのためのAWSの使い方入門

Slide 1

Slide 1 text

σʔλαΠΤϯεͷͨΊͷ AWSͷ࢖͍ํೖ໳ αϙʔλʔζCoLabษڧձ Jul 20 2017 ଜҪ ᠳଠ࿕

Slide 2

Slide 2 text

ࣗݾ঺հ • ଜҪᠳଠ࿕ • גࣜձࣾαΠόʔΤʔδΣϯτ • ΞυςΫຊ෦ ΠϯϑϥνʔϜ • ΠϯϑϥΤϯδχΞ ※ຊ೔ͷൃද͸ݸਓͷݟղͰ͋Γɺॴଐ͢Δ૊৫ͷެࣜݟղͰ͸͋Γ·ͤΜ

Slide 3

Slide 3 text

͓·͑AWSৄ͍͠ͷʁ

Slide 4

Slide 4 text

ਓฒΈʹ͸஌ͬͯΔ(͸ͣ) • AWSྺɿ3೥͘Β͍ • AWSͰ޷͖ͳαʔϏεɿRedshiftɺLambda • AWSೝఆɿAWS SAA • Re:Invent 2015 ࢀՃ͠·ͨ͠

Slide 5

Slide 5 text

ΊͪΌͪ͘Όৄ͋͘͠Γ·ͤΜ ઌ೔AWS SAPམͪ·ͨ͠…

Slide 6

Slide 6 text

օ༷ʹ࣭໰͍ͤͯͩ͘͞͞

Slide 7

Slide 7 text

σʔλαΠΤϯςΟετͷํ

Slide 8

Slide 8 text

σʔλ෼ੳ͸͠ͳ͍͚Ͳ  σʔλ෼ੳج൫ͷߏஙΛ͢Δํ

Slide 9

Slide 9 text

ͳΜͰͦ΋ͦ΋ࠓ೔ͷ࿩͠͸ σʔλαΠΤϯεͱAWSͳͷʁ

Slide 10

Slide 10 text

༑ਓ ࠷ۙɺσʔλαΠΤϯεςΟετʹͳͬͨΘ ͦΕ͸ͦΕ͸ɺ͓څྉ͗ΐ͏͞Μ΋͏ͯΔΜ΍Ζ ଜҪ ϕϯνϟʔ΍͔Β·ͩ·ͩ͜Ε͔Βɻ ·ͩGPUͱ͔αʔόʔΛങ͏͓ۚ΋ͳ͍͠ɻ Ͱ΋ֶशͤ͞Δͷʹ਺೔͔͔ΔͷΑͶʔ ͋ΕɺΫϥ΢υ࢖ͬͯͳ͍ͷʁ Ϋϥ΢υͬͯ࢖ͬͨ͜ͱͳ͍͚Ͳߴ͍Ͱ͠ΐ ͋ͱ࢖͏ͷ΋೉ͦ͠͏΍͠ ωοτϫʔΫͱ͔ߏஙͱ͔΋Ͱ͖ͳ͍͠ ͦΜͳ͜ͱ೉͍͜͠ͱ͋Β΁ΜͰ ͦ͏ͳͷʁ ΄ͳࠓ౓ίʔώʔ͓͝Δ͔Β͓͑ͯ͠΍ (130ԁͰ͸ͪΐͬͱ…)

Slide 11

Slide 11 text

ಉ྅ ػցֶश༻ͷαʔόʔͱ σʔλʹ෼ੳ༻ͷΫΤϦΛ౤͛ΔDBͱ ଞ෦ॺͱσʔλͷڞ༗΋͍ͨ͠  ͬͯݴΘΕͨͷ΍͚Ͳ૬ஊͤͯ͞ʔ ͓ɺ·͍Ͳʂ ͜Ε͸ͨͿΜΫϥ΢υҊ݅Ͱ͢Θ ଜҪ ͑ɺࣗࣾαʔόʔ͡Όͳ͘Ϋϥ΢υͳͷʁ ن໛ʹ΋ΑΓ·͕͢Ϋϥ΢υͷ΄͏͕  ։ൃεϐʔυ͕଎͘ɺίετ΋҆͘ɺ ӡ༻ָ͕Ͱ͢ͶΜ ΄͏΄͏ɺ΄ͳΘ͔ͬͯΔΜ΍ͬͨΒ  ͜ͷ͓ئ͍͍͍ͯ͠ʁ (͑…Զʹ΋λεΫ͕) ͜Ε͚ͩ΍ͱ΍Ε·΁Μɻ ҰճͪΌΜͱώΞϦϯάͤͯ͞΋Β͍·͢Θɻ

Slide 12

Slide 12 text

ͦΜͳਓ͸ੈͷதʹ ͪΐͬͱ͚͍ͩΔͷͰ͸ͳ͍͔ͱɻ

Slide 13

Slide 13 text

ຊ೔ͷର৅ऀ • Ϋϥ΢υ࢖͍͍ͨσʔλαΠΤϯςΟετͷํ • σʔλαΠΤϯςΟετ(Ά͍ਓ)ʹ  ج൫ͷߏஙΛ͓ئ͍͞Εͨํ • AWS࢖ͬͨ͜ͱͳ͍ํ • ࣗࣾ/ֶߍͰσʔλ෼ੳج൫Λ࣋ͬͯͳ͍ํ

Slide 14

Slide 14 text

ࠓ೔͓࿩͢Δ͜ͱ • AWSͱ͸ • σʔλαΠΤϯεͱ͸ • AWSαʔϏε঺հ • Amazon EC2 • Amazon EMR • Amazon S3 • Amazon Redshift • Ϋϥ΢υͰࣄނΛ๷͙ͨΊʹ

Slide 15

Slide 15 text

AWSͱ͸ʁ

Slide 16

Slide 16 text

AWS(Amazon Web Service) ͱ͸ Ϋϥ΢υίϯϐϡʔςΟϯάαʔϏε →Ϩϯλϧαʔόʔͷ͍͢͝΍ͭ ʮΠϯϑϥ͕WebαʔϏεʹͳͬͨʯ AWS ʹ͍ͭͯ https://aws.amazon.com/jp/about-aws/

Slide 17

Slide 17 text

AWS͸ੈքதʹ͋Δ • Ϧʔδϣϯ • େ཮ͳͲ஍ཧతʹ཭ΕͨྖҬ • 14ͭͷϦʔδϣϯ • ౦ژɺόʔδχΞɺϩϯυϯ • ΞϕΠϥϏϦςΟʔκʔϯ • 1 ͭͷϦʔδϣϯʹෳ਺ͷͦΕͧΕಠཱͨ͠ϩέʔγϣϯ • ྫ. ౦ژϦʔδϣϯͷAZɿ඼઒ɺཱ઒  ※࣮ࡍʹ͸ެ։͞Ε͍ͯ·ͤΜ EC2 ϦʔδϣϯͱΞϕΠϥϏϦςΟʔκʔϯ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-regions-availability-zones.html

Slide 18

Slide 18 text

AWSͰग़དྷΔ͜ͱ(΄ΜͷҰ෦) • ؆୯ʹߏஙͰ͖Δίϯϐϡʔτ(αʔόʔ) • EC2 • ΫϦοΫ͚ͩͰग़དྷΔσʔλϕʔεͷߏங • RDS • ੍ݶແ͠ͷσʔλετϨʔδ • S3 • ετϦʔϜॲཧ • Kinesis (Stream)

Slide 19

Slide 19 text

σʔλαΠΤϯεͱ͸ʁ σʔλΛऩूɺ෼ੳɺར༻ͯ͠༗ޮ׆༻͢Δٕज़·ͨ͸ֶ໰

Slide 20

Slide 20 text

σʔλαΠΤϯςΟετͱ͸ σʔλαΠΤϯεྗɺ  σʔλΤϯδχΞϦϯάྗΛϕʔεʹ  σʔλ͔ΒՁ஋Λ૑ग़͠ɺ  Ϗδωε՝୊ʹ౴͑Λग़͢  ϓϩϑΣογϣφϧ  ※͜͜ͰʮϏδωεʯͱ͸ࣾձʹ໾ʹཱͭҙຯͷ͋Δ׆ಈશൠΛࢦ͢ Ұൠࣾஂ๏ਓσʔλαΠΤϯςΟετڠձ σʔλαΠΤϯςΟετͷϛογϣϯɺεΩϧηοτɺఆٛɺεΩϧϨϕϧΛൃද http://www.datascientist.or.jp/news/2014/pdf/1210.pdf

Slide 21

Slide 21 text

ͳͥσʔλαΠΤϯεͰ AWS(Ϋϥ΢υ)Λ࢖͏ͷ͔

Slide 22

Slide 22 text

σʔλ෼ੳʹٻΊΒΕΔ͜ͱ • σʔλΞφϦςΟΫε • େ͖͍σʔλΛ҆Ձʹѻ͍͍ͨ • σʔλ΢ΣΞϋ΢εʹ෼ੳΫΤϦΛ࣮ߦ͍ͨ͠ • ؆୯ʹՄࢹԽ͍ͨ͠ • ػցֶश࣮ߦج൫ • ֶश࣌”͚ͩ”ඞཁʹͳΔେྔͷGPU • ӡ༻ΛͳΔ΂ͨ͘͘͠ͳ͍ • ߏ੒ͳͲࣗ༝౓͕ߴ͍

Slide 23

Slide 23 text

ඞཁͳ࣌ʹ ඞཁͳϦιʔε͚ͩΛ֬อ͠ ඞཁͳ෼͚ͩ෼ੳ͍ͨ͠ Ϋϥ΢υ͕࠷దͰ͢Αʂ

Slide 24

Slide 24 text

σʔλ෼ੳ͢Δࡍʹ  AWSͷͲͷαʔϏε͕࢖͑Δ͔

Slide 25

Slide 25 text

঺հ͢ΔαʔϏε Amazon EC2 Amazon S3 Amazon EMR Amazon Redshift Ծ૝αʔόߏஙαʔϏε εέʔϥϒϧͰ ࢖ͬͨ෼͚ͩͷैྔ՝ۚ ϏοάσʔλϑϨʔϜϫʔΫΛ ؆୯ʹߏஙͰ͖ΔϚωʔδυαʔϏε HadoopɺSparkΫϥελʔ͕؆୯ʹߏஙͰ͖Δ σʔλετϨʔδαʔϏε ແ੍ݶʹσʔλ͕อ؅Ͱ͖Δ ੩తίϯςϯπͷWebϗεςΟϯά΋Մೳ AWS͕ఏڙ͢Δ ׬શϚωʔδυͷσʔλ΢ΣΞϋ΢ε SQL͕࣮ߦͰ͖ϖλόΠτن໛ͷ σʔλΛѻ͑Δ

Slide 26

Slide 26 text

Amazon EC2

Slide 27

Slide 27 text

Amazon EC2ͱ͸ • Ϋϥ΢υίϯϐϡʔςΟϯά • ͍ΘΏΔԾ૝αʔόͷߏங • ྉۚମܥ • ίϯϐϡʔτͷىಈ࣌ؒʹԠͯ͡՝ۚ • ϘϦϡʔϜʹର͢Δ՝ۚ • ωοτϫʔΫసૹྔʹର͢Δ՝ۚ • ๛෋ͳΠϯελϯελΠϓ Amazon EC2 ͱ͸ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/concepts.html

Slide 28

Slide 28 text

Amazon EC2ͷ஫ҙ఺ • ॳظ͸όʔδχΞ๺෦ϦʔδϣϯͰىಈ • ηΩϡϦςΟάϧʔϓͰϙʔτΛղ์͠ͳ͍ͱ  ωοτϫʔΫͷૄ௨͕ग़དྷͳ͍ • ηΩϡϦςΟάϧʔϓͰશެ։[0.0.0.0/0]Λ  ઃఆ͢Δࡍ͸஫ҙΛʂ • ύϒϦοΫIP/ElasticIPΛ෇༩͢Δඞཁ͋Γ  (VPNͳͲ઀ଓͯ͠Δ৔߹Λআ͘) • ՝ۚ͸1࣌ؒ୯Ґ • 1෼࢖ͬͯ΋ɺ59෼࢖ͬͯ΋ಉ͡ྉۚ

Slide 29

Slide 29 text

EC2εϙοτΠϯελϯε • Amazonʹམͱ͞ΕΔ͔΋͠Εͳ͍͚Ͳɺ௿Ձ֨ͳΠϯελϯε • ೖࡳՁ֨ > ࢢ৔Ձ֨ͳΒࢢ৔Ձ֨ͰΠϯελϯεΛىಈͰ͖Δ • ࢢ৔Ձ֨ > ೖࡳՁ֨ͱͳΔͱΠϯελϯε͸ऴྃ • 70%off͘Β͍Ͱ࢖͑Δ(ݸਓ࣮੷)  ※ΦϯσϚϯυΠϯελϯεͱͷൺֱ • ೖࡳΞυόΠβʔΛ࢖͏  https://aws.amazon.com/jp/ec2/spot/bid-advisor/ Amazon EC2 εϙοτΠϯελϯε http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-spot-instances.html

Slide 30

Slide 30 text

ͱ͜ΖͰػցֶश͍ͨ͠ͷͰ GPUΛ࢖͍͍ͨ

Slide 31

Slide 31 text

GPUͬͯͳʹʁ • Graphics Processing Unit • ݩʑɺը૾ॲཧͳͲ޲͚ʹ։ൃ͞Εͨ • ฒྻԋࢉ͕ಘҙͳϓϩηοα • ߦྻԋࢉ͕ಘҙ  ˠ σΟʔϓϥʔχϯάʹద͍ͯ͠Δ (16 $16

Slide 32

Slide 32 text

GPU࢖͏ͱͳͥ଎͍ͷ͔ • ԋࢉग़དྷΔίΞ਺͕2000-4000ίΞ • ฒྻܭࢉॲཧʹ༗ར • CPU(Intel Xeonͱ͔)ͩͱ࠷େ48ίΞ͘Β͍ • ϝϞϦ͕ಠཱ͔ͭߴ଎ಡΈࠐΈ • CPUͷϝϞϦόϯυ෯ΑΓ10ഒ͘Β͍଎͍ • σΟʔϓϥʔχϯάͰ΋ར༻͢Δߦྻܭࢉ  Λ௒ฒྻͰߦ͑ΔͷͰAIج൫Ͱ΋ར༻͕ਐΉ

Slide 33

Slide 33 text

ৄ͘͠GPUͷ͍ͭͯ஌Γ͍ͨํ͸ ͜ͷຊ͕͓͢͢ΊͰ͢ GPUΛࢧ͑Δٕज़  ஶऀɿHisa Ando ൃച೔ɿ2017೥6݄30೔ URLɿhttps://www.amazon.co.jp/dp/477419056X ※ΞϑΟϦϯΫͰ͸͋Γ·ͤΜ

Slide 34

Slide 34 text

AWSͰ࢖͑ΔGPU

Slide 35

Slide 35 text

GPUΠϯελϯελΠϓ p2Πϯελϯε g2 Πϯελϯε g3Πϯελϯε ←new

Slide 36

Slide 36 text

౥ࡌGPUΠϯελϯε Πϯελϯε $16 W$16 ϝϞϦ (16 (16ϝϞϦ උߟ QYMBSHF (# /7*%*"5FTMB, (# QYMBSHF (# (# QYMBSHF (# (# HYMBSHF (# /7*%*"(3*%, (# HYMBSHF (# (# HYMBSHF (# /7*%*"5FTMB. (# ౦ژϦʔδϣϯ Ͱ͸࢖͑ͳ͍ HYMBSHF (# (# HYMBSHF (# (# ※vCPU͸εϨου਺

Slide 37

Slide 37 text

Amazon EMR

Slide 38

Slide 38 text

Amazon EMRͱ͸ • EC2Πϯελϯε্ʹ෼ࢄॲཧج൫Λ  ΫϥελʔͰߏங͢ΔαʔϏε • HadoopɺApache Spark؀ڥͳͲ

Slide 39

Slide 39 text

Amazon EMRͷ஫ҙ఺ͳͲ • S3ʹ͋ΔΞϓϦέʔγϣϯΛࢦఆ࣮ͯ͠ߦͰ ͖ΔͷͰྑ͍ • ϩά΋S3ʹग़ྗͰ͖ΔͷͰϩά͕อଘग़དྷΔ • ؂ࢹ͕೉͍͠ • ΞϓϦέʔγϣϯଆͰৄࡉͳΤϥʔΛग़ྗͨ͠΄͏͕͍͍ • EMRͰ͸੒ޭ/ࣦഊ͔͠ग़ྗ͞Εͳ͍

Slide 40

Slide 40 text

Amazon S3

Slide 41

Slide 41 text

σʔλϨΠΫ • σʔλϨΠΫͱ͸ • ੜϩάɺը૾ɾө૾ɺԻ੠ɺͳͲͷ  ʮඇߏ଄ԽσʔλʯΛ؅ཧ͢ΔྖҬ • ඞཁͳ࣌ʹσʔλϨΠΫ͔ΒऔΓग़͠ɺ  ͙͢͞·σʔλΛ׆༻Ͱ͖Δ GE Announces First Data Lake Approach for Industrial Internet to Better Access, Analyze and Store Industrial-Strength Big Data http://www.businesswire.com/news/home/20140810005024/en/GE-Announces-Data-Lake-Approach-Industrial-Internet

Slide 42

Slide 42 text

Amazon S3 • AWSͷετϨʔδαʔϏε  σʔλϨΠΫͱͯ͠΋࢖͑Δ • ಛ௃ • ଱ٱੑ 99.999999999% • 1ສݸͷΦϒδΣΫτͷ͏ͪ1ͭͷফࣦʹ1000ສ೥͔͔Δ • Մ༻ੑ 99.99% • S3αʔϏεμ΢ϯλΠϜ 52෼34ඵ • ࡉ͔͘ઃఆͰ͖ΔΞΫηείϯτϩʔϧ • ACLɺόέοτϙϦγʔɺAIMϙϦγʔ S3ͷΞΫηείϯτϩʔϧ·ͱΊ http://qiita.com/ryo0301/items/791c0a666feeea0a704c

Slide 43

Slide 43 text

Amazon S3ͷछྨ • S3 ඪ४ • ී௨ͷετϨʔδ • S3 ඪ४ – ௿৑௕ԽετϨʔδ • ॏཁੑͷ௿͍࠶ੜՄೳͳσʔλ޲͚ • S3 ඪ४ – ௿ස౓ΞΫηε • ΞΫηε͕গͳ͍σʔλ޲͚ • ΞʔΧΠϒ(Amazon Glacier) • ΊͬͨʹΞΫηε͞Εͳ͍σʔλ༻ https://aws.amazon.com/jp/s3/storage-classes/

Slide 44

Slide 44 text

Amazon S3ͷྉۚମܥ • ετϨʔδྉۚ • ͲΕ͘Β͍σʔλྔΛஔ͍ͯΔ͔ • σʔλసૹྔ • ͲΕ͘Β͍σʔλΛ࣋ͪग़͔ͨ͠ • APIίʔϧྉۚ • Ξοϓϩʔυ/࡟আ/μ΢ϯϩʔυͳͲͷૢ࡞

Slide 45

Slide 45 text

Amazon S3ͷ஫ҙ఺ • σʔλసૹྔ͕݁ߏߴ͍Ͱ͢ ex.1TB(1024GB)ΛΠϯλʔωοτܦ༝Ͱ  μ΢ϯϩʔυͨ͠Β໿1.5ສԁ($143.36) • σϑΥϧτͷอ؅ઌ͕  ถࠃ౦෦(όʔδχΞ)

Slide 46

Slide 46 text

Amazon Redshift

Slide 47

Slide 47 text

Amazon Redshiftͱ͸ • AWS͕ఏڙ͢Δ׬શϚωʔδυܕͷ  σʔλ΢ΣΞϋ΢ε(DHW) • ϖλόΠτΫϥεʹεέʔϧ͢Δ • ྻࢦ޲σʔλϕʔε • ྉۚମܥ͸ΦϯσϚϯυྉۚ

Slide 48

Slide 48 text

Amazon Redshiftͷ஫ҙ఺ • σʔλߏ଄ͷઃܭΛ͢Δ(͋ͨΓલ?) • ಛʹ෼ࢄΩʔɺιʔτΩʔɺ࣌ܥྻςʔϒϧ෼ׂ • ࣌ʑɺڧ੍ϝϯςφϯε͕ൃੜ͢Δ • ϦΞϧλΠϜॲཧʹ͸ෆ޲͖ • ͋͘·Ͱ෼ੳ༻ʹ࢖͏ • ຊ൪؀ڥͰར༻͢Δ৔߹͸μ΢ϯλΠϜΛ૝ఆ͢Δ • μογϡϘʔυ͸༏ल

Slide 49

Slide 49 text

঺հ͔ͨͬͨ͠αʔϏε Amazon RDS AWS͕ఏڙ͢ΔϚωʔδυܕ ϦϨʔγϣφϧσʔλϕʔε Amazon Athena Amazon S3 ಺ͷσʔλʹ SQL Λ࣮ߦͰ͖ΔαʔϏε 6݄ʹ౦ژϦʔδϣϯ։࢝ Amazon Kinesis Amazon Elasticsearch AWS Lambda Amazon VPC Amazon CloudWatch Amazonͷఏڙ͢Δ Ծ૝ωοτϫʔΫαʔϏε جຊ͸ແྉͰར༻Մೳ ετϦʔϛϯάॲཧ͕ ग़དྷΔϚωʔδυαʔϏε αʔόʔϨεͰϓϩάϥϜΛ ࣮ߦͰ͖ΔαʔϏε ElasticsearchΛΤϯδϯ ͱͨ͠ϚωʔδυαʔϏε AWSͷ؂ࢹπʔϧ

Slide 50

Slide 50 text

Ϋϥ΢υͰࣄނΛ๷͙ͨΊʹ

Slide 51

Slide 51 text

ઈର΍Δ΂͖͜ͱ • ྉۚΞϥʔϜΛઃఆ͢Δ  Ξϥʔτͱ௨஌Ͱ੥ٻֹΛϞχλϦϯά  http://docs.aws.amazon.com/ja_jp/awsaccountbilling/latest/aboutv2/monitor-charges.html • APIΩʔͷ؅ཧ͸ެ։ઈର͠ͳ͍ • Access key/Secret Key͸ඞͣެ։͠ͳ͍Α͏ʹ • ʮॳ৺ऀ͕AWSͰϛεͬͯෆਖ਼ར༻͞Εͯ$6,000੥ٻɺٽ͖ͦ͏ʹͳ͓ͬͨ࿩ʯͳͲ • ݟੵ΋Γ͸ඞ͓ͣ͜ͳ͏ • AWS ؆қݟੵΓπʔϧ  http://calculator.s3.amazonaws.com/index.html?lng=ja_JP • AWS ՝ۚମܥͱݟੵΓํ๏ʹ͍ͭͯ  https://aws.amazon.com/jp/how-to-understand-pricing/ • ηΩϡϦςΟάϧʔϓ͸[ssh 0.0.0.0/0]ͱ͔ઃఆ͠ͳ͍ • ݁ߏϒϧʔτϑΥʔε(૯౰Γ߈ܸ)ͱ͔ɺ੬ऑੑ߈ܸ͕͋Γ·͢

Slide 52

Slide 52 text

࣭໰λΠϜ

Slide 53

Slide 53 text

AWS΍σʔλαΠΤϯε ΍Γ͍ͨͻͱ ϚδͰઈࢍืूத ※ΦϑΟεݟֶ͚ͩͰ΋େৎ෉Ͱ͢ αΠόʔΤʔδΣϯτ ΞυςΫελδΦ https://adtech.cyberagent.io/

Slide 54

Slide 54 text

͋Γ͕ͱ͏͍͟͝·ͨ͠

Slide 55

Slide 55 text

ࢀߟ৘ใ AWS υΩϡϝϯτ https://aws.amazon.com/jp/documentation/ AWS Ϋϥ΢υαʔϏε׆༻ࢿྉू https://aws.amazon.com/jp/aws-jp-introduction/ Youtube AWSެࣜνϟϯωϧ(Re:InventͷηογϣϯͳͲ) ӳޠ https://www.youtube.com/user/AmazonWebServices ೔ຊޠ https://www.youtube.com/user/AmazonWebServicesJP  AWSΫϥ΢υσβΠϯύλʔϯ http://aws.clouddesignpattern.org/index.phpύλʔϯ Developers.IO http://dev.classmethod.jp/ DeNA TechCon 2017 ͱ Developers Summit 2017 ͰDeNAͷػցֶशج൫ͱ෼ੳج൫ͷߨԋΛ͠·ͨ͠ http://blog.livedoor.jp/sonots/archives/49502478.html