データサイエンスのためのAWSの使い方入門

Transcript

1. σʔλαΠΤϯεͷͨΊͷ AWSͷ࢖͍ํೖ໳ αϙʔλʔζCoLabษڧձ Jul 20 2017 ଜҪ ᠳଠ࿕

2. ࣗݾ঺հ • ଜҪᠳଠ࿕ • גࣜձࣾαΠόʔΤʔδΣϯτ • ΞυςΫຊ෦ ΠϯϑϥνʔϜ • ΠϯϑϥΤϯδχΞ

   ※ຊ೔ͷൃද͸ݸਓͷݟղͰ͋Γɺॴଐ͢Δ૊৫ͷެࣜݟղͰ͸͋Γ·ͤΜ

3. ͓·͑AWSৄ͍͠ͷʁ

4. ਓฒΈʹ͸஌ͬͯΔ(͸ͣ) • AWSྺɿ3೥͘Β͍ • AWSͰ޷͖ͳαʔϏεɿRedshiftɺLambda • AWSೝఆɿAWS SAA • Re:Invent

   2015 ࢀՃ͠·ͨ͠

5. ΊͪΌͪ͘Όৄ͋͘͠Γ·ͤΜ ઌ೔AWS SAPམͪ·ͨ͠…

6. օ༷ʹ࣭໰͍ͤͯͩ͘͞͞

7. σʔλαΠΤϯςΟετͷํ

8. σʔλ෼ੳ͸͠ͳ͍͚Ͳ
 σʔλ෼ੳج൫ͷߏஙΛ͢Δํ

9. ͳΜͰͦ΋ͦ΋ࠓ೔ͷ࿩͠͸ σʔλαΠΤϯεͱAWSͳͷʁ

10. ༑ਓ ࠷ۙɺσʔλαΠΤϯεςΟετʹͳͬͨΘ ͦΕ͸ͦΕ͸ɺ͓څྉ͗ΐ͏͞Μ΋͏ͯΔΜ΍Ζ ଜҪ ϕϯνϟʔ΍͔Β·ͩ·ͩ͜Ε͔Βɻ ·ͩGPUͱ͔αʔόʔΛങ͏͓ۚ΋ͳ͍͠ɻ Ͱ΋ֶशͤ͞Δͷʹ਺೔͔͔ΔͷΑͶʔ ͋ΕɺΫϥ΢υ࢖ͬͯͳ͍ͷʁ Ϋϥ΢υͬͯ࢖ͬͨ͜ͱͳ͍͚Ͳߴ͍Ͱ͠ΐ ͋ͱ࢖͏ͷ΋೉ͦ͠͏΍͠

    ωοτϫʔΫͱ͔ߏஙͱ͔΋Ͱ͖ͳ͍͠ ͦΜͳ͜ͱ೉͍͜͠ͱ͋Β΁ΜͰ ͦ͏ͳͷʁ ΄ͳࠓ౓ίʔώʔ͓͝Δ͔Β͓͑ͯ͠΍ (130ԁͰ͸ͪΐͬͱ…)

11. ಉ྅ ػցֶश༻ͷαʔόʔͱ σʔλʹ෼ੳ༻ͷΫΤϦΛ౤͛ΔDBͱ ଞ෦ॺͱσʔλͷڞ༗΋͍ͨ͠
 ͬͯݴΘΕͨͷ΍͚Ͳ૬ஊͤͯ͞ʔ ͓ɺ·͍Ͳʂ ͜Ε͸ͨͿΜΫϥ΢υҊ݅Ͱ͢Θ ଜҪ ͑ɺࣗࣾαʔόʔ͡Όͳ͘Ϋϥ΢υͳͷʁ ن໛ʹ΋ΑΓ·͕͢Ϋϥ΢υͷ΄͏͕


    ։ൃεϐʔυ͕଎͘ɺίετ΋҆͘ɺ ӡ༻ָ͕Ͱ͢ͶΜ ΄͏΄͏ɺ΄ͳΘ͔ͬͯΔΜ΍ͬͨΒ
 ͜ͷ͓ئ͍͍͍ͯ͠ʁ (͑…Զʹ΋λεΫ͕) ͜Ε͚ͩ΍ͱ΍Ε·΁Μɻ ҰճͪΌΜͱώΞϦϯάͤͯ͞΋Β͍·͢Θɻ

12. ͦΜͳਓ͸ੈͷதʹ ͪΐͬͱ͚͍ͩΔͷͰ͸ͳ͍͔ͱɻ

13. ຊ೔ͷର৅ऀ • Ϋϥ΢υ࢖͍͍ͨσʔλαΠΤϯςΟετͷํ • σʔλαΠΤϯςΟετ(Ά͍ਓ)ʹ
 ج൫ͷߏஙΛ͓ئ͍͞Εͨํ • AWS࢖ͬͨ͜ͱͳ͍ํ • ࣗࣾ/ֶߍͰσʔλ෼ੳج൫Λ࣋ͬͯͳ͍ํ

14. ࠓ೔͓࿩͢Δ͜ͱ • AWSͱ͸ • σʔλαΠΤϯεͱ͸ • AWSαʔϏε঺հ • Amazon EC2

    • Amazon EMR • Amazon S3 • Amazon Redshift • Ϋϥ΢υͰࣄނΛ๷͙ͨΊʹ

15. AWSͱ͸ʁ

16. AWS(Amazon Web Service) ͱ͸ Ϋϥ΢υίϯϐϡʔςΟϯάαʔϏε →Ϩϯλϧαʔόʔͷ͍͢͝΍ͭ ʮΠϯϑϥ͕WebαʔϏεʹͳͬͨʯ AWS ʹ͍ͭͯ https://aws.amazon.com/jp/about-aws/

17. AWS͸ੈքதʹ͋Δ • Ϧʔδϣϯ • େ཮ͳͲ஍ཧతʹ཭ΕͨྖҬ • 14ͭͷϦʔδϣϯ • ౦ژɺόʔδχΞɺϩϯυϯ •

    ΞϕΠϥϏϦςΟʔκʔϯ • 1 ͭͷϦʔδϣϯʹෳ਺ͷͦΕͧΕಠཱͨ͠ϩέʔγϣϯ • ྫ. ౦ژϦʔδϣϯͷAZɿ඼઒ɺཱ઒
 ※࣮ࡍʹ͸ެ։͞Ε͍ͯ·ͤΜ EC2 ϦʔδϣϯͱΞϕΠϥϏϦςΟʔκʔϯ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-regions-availability-zones.html

18. AWSͰग़དྷΔ͜ͱ(΄ΜͷҰ෦) • ؆୯ʹߏஙͰ͖Δίϯϐϡʔτ(αʔόʔ) • EC2 • ΫϦοΫ͚ͩͰग़དྷΔσʔλϕʔεͷߏங • RDS •

    ੍ݶແ͠ͷσʔλετϨʔδ • S3 • ετϦʔϜॲཧ • Kinesis (Stream)

19. σʔλαΠΤϯεͱ͸ʁ σʔλΛऩूɺ෼ੳɺར༻ͯ͠༗ޮ׆༻͢Δٕज़·ͨ͸ֶ໰

20. σʔλαΠΤϯςΟετͱ͸ σʔλαΠΤϯεྗɺ
 σʔλΤϯδχΞϦϯάྗΛϕʔεʹ
 σʔλ͔ΒՁ஋Λ૑ग़͠ɺ
 Ϗδωε՝୊ʹ౴͑Λग़͢
 ϓϩϑΣογϣφϧ
 ※͜͜ͰʮϏδωεʯͱ͸ࣾձʹ໾ʹཱͭҙຯͷ͋Δ׆ಈશൠΛࢦ͢ Ұൠࣾஂ๏ਓσʔλαΠΤϯςΟετڠձ σʔλαΠΤϯςΟετͷϛογϣϯɺεΩϧηοτɺఆٛɺεΩϧϨϕϧΛൃද http://www.datascientist.or.jp/news/2014/pdf/1210.pdf

21. ͳͥσʔλαΠΤϯεͰ AWS(Ϋϥ΢υ)Λ࢖͏ͷ͔

22. σʔλ෼ੳʹٻΊΒΕΔ͜ͱ • σʔλΞφϦςΟΫε • େ͖͍σʔλΛ҆Ձʹѻ͍͍ͨ • σʔλ΢ΣΞϋ΢εʹ෼ੳΫΤϦΛ࣮ߦ͍ͨ͠ • ؆୯ʹՄࢹԽ͍ͨ͠ •

    ػցֶश࣮ߦج൫ • ֶश࣌”͚ͩ”ඞཁʹͳΔେྔͷGPU • ӡ༻ΛͳΔ΂ͨ͘͘͠ͳ͍ • ߏ੒ͳͲࣗ༝౓͕ߴ͍

23. ඞཁͳ࣌ʹ ඞཁͳϦιʔε͚ͩΛ֬อ͠ ඞཁͳ෼͚ͩ෼ੳ͍ͨ͠ Ϋϥ΢υ͕࠷దͰ͢Αʂ

24. σʔλ෼ੳ͢Δࡍʹ
 AWSͷͲͷαʔϏε͕࢖͑Δ͔

25. ঺հ͢ΔαʔϏε Amazon EC2 Amazon S3 Amazon EMR Amazon Redshift Ծ૝αʔόߏஙαʔϏε

    εέʔϥϒϧͰ ࢖ͬͨ෼͚ͩͷैྔ՝ۚ ϏοάσʔλϑϨʔϜϫʔΫΛ ؆୯ʹߏஙͰ͖ΔϚωʔδυαʔϏε HadoopɺSparkΫϥελʔ͕؆୯ʹߏஙͰ͖Δ σʔλετϨʔδαʔϏε ແ੍ݶʹσʔλ͕อ؅Ͱ͖Δ ੩తίϯςϯπͷWebϗεςΟϯά΋Մೳ AWS͕ఏڙ͢Δ ׬શϚωʔδυͷσʔλ΢ΣΞϋ΢ε SQL͕࣮ߦͰ͖ϖλόΠτن໛ͷ σʔλΛѻ͑Δ

26. Amazon EC2

27. Amazon EC2ͱ͸ • Ϋϥ΢υίϯϐϡʔςΟϯά • ͍ΘΏΔԾ૝αʔόͷߏங • ྉۚମܥ • ίϯϐϡʔτͷىಈ࣌ؒʹԠͯ͡՝ۚ

    • ϘϦϡʔϜʹର͢Δ՝ۚ • ωοτϫʔΫసૹྔʹର͢Δ՝ۚ • ๛෋ͳΠϯελϯελΠϓ Amazon EC2 ͱ͸ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/concepts.html

28. Amazon EC2ͷ஫ҙ఺ • ॳظ͸όʔδχΞ๺෦ϦʔδϣϯͰىಈ • ηΩϡϦςΟάϧʔϓͰϙʔτΛղ์͠ͳ͍ͱ
 ωοτϫʔΫͷૄ௨͕ग़དྷͳ͍ • ηΩϡϦςΟάϧʔϓͰશެ։[0.0.0.0/0]Λ
 ઃఆ͢Δࡍ͸஫ҙΛʂ

    • ύϒϦοΫIP/ElasticIPΛ෇༩͢Δඞཁ͋Γ
 (VPNͳͲ઀ଓͯ͠Δ৔߹Λআ͘) • ՝ۚ͸1࣌ؒ୯Ґ • 1෼࢖ͬͯ΋ɺ59෼࢖ͬͯ΋ಉ͡ྉۚ

29. EC2εϙοτΠϯελϯε • Amazonʹམͱ͞ΕΔ͔΋͠Εͳ͍͚Ͳɺ௿Ձ֨ͳΠϯελϯε • ೖࡳՁ֨ > ࢢ৔Ձ֨ͳΒࢢ৔Ձ֨ͰΠϯελϯεΛىಈͰ͖Δ • ࢢ৔Ձ֨ >

    ೖࡳՁ֨ͱͳΔͱΠϯελϯε͸ऴྃ • 70%off͘Β͍Ͱ࢖͑Δ(ݸਓ࣮੷)
 ※ΦϯσϚϯυΠϯελϯεͱͷൺֱ • ೖࡳΞυόΠβʔΛ࢖͏
 https://aws.amazon.com/jp/ec2/spot/bid-advisor/ Amazon EC2 εϙοτΠϯελϯε http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-spot-instances.html

30. ͱ͜ΖͰػցֶश͍ͨ͠ͷͰ GPUΛ࢖͍͍ͨ

31. GPUͬͯͳʹʁ • Graphics Processing Unit • ݩʑɺը૾ॲཧͳͲ޲͚ʹ։ൃ͞Εͨ • ฒྻԋࢉ͕ಘҙͳϓϩηοα •

    ߦྻԋࢉ͕ಘҙ
 ˠ σΟʔϓϥʔχϯάʹద͍ͯ͠Δ (16 $16

32. GPU࢖͏ͱͳͥ଎͍ͷ͔ • ԋࢉग़དྷΔίΞ਺͕2000-4000ίΞ • ฒྻܭࢉॲཧʹ༗ར • CPU(Intel Xeonͱ͔)ͩͱ࠷େ48ίΞ͘Β͍ • ϝϞϦ͕ಠཱ͔ͭߴ଎ಡΈࠐΈ

    • CPUͷϝϞϦόϯυ෯ΑΓ10ഒ͘Β͍଎͍ • σΟʔϓϥʔχϯάͰ΋ར༻͢Δߦྻܭࢉ
 Λ௒ฒྻͰߦ͑ΔͷͰAIج൫Ͱ΋ར༻͕ਐΉ

33. ৄ͘͠GPUͷ͍ͭͯ஌Γ͍ͨํ͸ ͜ͷຊ͕͓͢͢ΊͰ͢ GPUΛࢧ͑Δٕज़
 ஶऀɿHisa Ando ൃച೔ɿ2017೥6݄30೔ URLɿhttps://www.amazon.co.jp/dp/477419056X ※ΞϑΟϦϯΫͰ͸͋Γ·ͤΜ

34. AWSͰ࢖͑ΔGPU

35. GPUΠϯελϯελΠϓ p2Πϯελϯε g2 Πϯελϯε g3Πϯελϯε ←new

36. ౥ࡌGPUΠϯελϯε Πϯελϯε $16 W$16 ϝϞϦ (16 (16ϝϞϦ උߟ QYMBSHF 

     (# /7*%*"
    5FTMB
    , (# QYMBSHF   (# (# QYMBSHF   (# (# HYMBSHF   (# /7*%*"
    (3*%
    , (# HYMBSHF   (# (# HYMBSHF   (# /7*%*"
    5FTMB
    . (# ౦ژϦʔδϣϯ
    Ͱ͸࢖͑ͳ͍ HYMBSHF   (# (# HYMBSHF   (# (# ※vCPU͸εϨου਺

37. Amazon EMR

38. Amazon EMRͱ͸ • EC2Πϯελϯε্ʹ෼ࢄॲཧج൫Λ
 ΫϥελʔͰߏங͢ΔαʔϏε • HadoopɺApache Spark؀ڥͳͲ

39. Amazon EMRͷ஫ҙ఺ͳͲ • S3ʹ͋ΔΞϓϦέʔγϣϯΛࢦఆ࣮ͯ͠ߦͰ ͖ΔͷͰྑ͍ • ϩά΋S3ʹग़ྗͰ͖ΔͷͰϩά͕อଘग़དྷΔ • ؂ࢹ͕೉͍͠ •

    ΞϓϦέʔγϣϯଆͰৄࡉͳΤϥʔΛग़ྗͨ͠΄͏͕͍͍ • EMRͰ͸੒ޭ/ࣦഊ͔͠ग़ྗ͞Εͳ͍

40. Amazon S3

41. σʔλϨΠΫ • σʔλϨΠΫͱ͸ • ੜϩάɺը૾ɾө૾ɺԻ੠ɺͳͲͷ
 ʮඇߏ଄ԽσʔλʯΛ؅ཧ͢ΔྖҬ • ඞཁͳ࣌ʹσʔλϨΠΫ͔ΒऔΓग़͠ɺ
 ͙͢͞·σʔλΛ׆༻Ͱ͖Δ GE

    Announces First Data Lake Approach for Industrial Internet to Better Access, Analyze and Store Industrial-Strength Big Data http://www.businesswire.com/news/home/20140810005024/en/GE-Announces-Data-Lake-Approach-Industrial-Internet

42. Amazon S3 • AWSͷετϨʔδαʔϏε
 σʔλϨΠΫͱͯ͠΋࢖͑Δ • ಛ௃ • ଱ٱੑ 99.999999999%

    • 1ສݸͷΦϒδΣΫτͷ͏ͪ1ͭͷফࣦʹ1000ສ೥͔͔Δ • Մ༻ੑ 99.99% • S3αʔϏεμ΢ϯλΠϜ 52෼34ඵ • ࡉ͔͘ઃఆͰ͖ΔΞΫηείϯτϩʔϧ • ACLɺόέοτϙϦγʔɺAIMϙϦγʔ S3ͷΞΫηείϯτϩʔϧ·ͱΊ http://qiita.com/ryo0301/items/791c0a666feeea0a704c

43. Amazon S3ͷछྨ • S3 ඪ४ • ී௨ͷετϨʔδ • S3 ඪ४

    – ௿৑௕ԽετϨʔδ • ॏཁੑͷ௿͍࠶ੜՄೳͳσʔλ޲͚ • S3 ඪ४ – ௿ස౓ΞΫηε • ΞΫηε͕গͳ͍σʔλ޲͚ • ΞʔΧΠϒ(Amazon Glacier) • ΊͬͨʹΞΫηε͞Εͳ͍σʔλ༻ https://aws.amazon.com/jp/s3/storage-classes/

44. Amazon S3ͷྉۚମܥ • ετϨʔδྉۚ • ͲΕ͘Β͍σʔλྔΛஔ͍ͯΔ͔ • σʔλసૹྔ • ͲΕ͘Β͍σʔλΛ࣋ͪग़͔ͨ͠

    • APIίʔϧྉۚ • Ξοϓϩʔυ/࡟আ/μ΢ϯϩʔυͳͲͷૢ࡞

45. Amazon S3ͷ஫ҙ఺ • σʔλసૹྔ͕݁ߏߴ͍Ͱ͢ ex.1TB(1024GB)ΛΠϯλʔωοτܦ༝Ͱ
 μ΢ϯϩʔυͨ͠Β໿1.5ສԁ($143.36) • σϑΥϧτͷอ؅ઌ͕
 ถࠃ౦෦(όʔδχΞ)

46. Amazon Redshift

47. Amazon Redshiftͱ͸ • AWS͕ఏڙ͢Δ׬શϚωʔδυܕͷ
 σʔλ΢ΣΞϋ΢ε(DHW) • ϖλόΠτΫϥεʹεέʔϧ͢Δ • ྻࢦ޲σʔλϕʔε •

    ྉۚମܥ͸ΦϯσϚϯυྉۚ

48. Amazon Redshiftͷ஫ҙ఺ • σʔλߏ଄ͷઃܭΛ͢Δ(͋ͨΓલ?) • ಛʹ෼ࢄΩʔɺιʔτΩʔɺ࣌ܥྻςʔϒϧ෼ׂ • ࣌ʑɺڧ੍ϝϯςφϯε͕ൃੜ͢Δ • ϦΞϧλΠϜॲཧʹ͸ෆ޲͖

    • ͋͘·Ͱ෼ੳ༻ʹ࢖͏ • ຊ൪؀ڥͰར༻͢Δ৔߹͸μ΢ϯλΠϜΛ૝ఆ͢Δ • μογϡϘʔυ͸༏ल

49. ঺հ͔ͨͬͨ͠αʔϏε Amazon RDS AWS͕ఏڙ͢ΔϚωʔδυܕ ϦϨʔγϣφϧσʔλϕʔε Amazon Athena Amazon S3 ಺ͷσʔλʹ

    SQL Λ࣮ߦͰ͖ΔαʔϏε 6݄ʹ౦ژϦʔδϣϯ։࢝ Amazon Kinesis Amazon Elasticsearch AWS Lambda Amazon VPC Amazon CloudWatch Amazonͷఏڙ͢Δ Ծ૝ωοτϫʔΫαʔϏε جຊ͸ແྉͰར༻Մೳ ετϦʔϛϯάॲཧ͕ ग़དྷΔϚωʔδυαʔϏε αʔόʔϨεͰϓϩάϥϜΛ ࣮ߦͰ͖ΔαʔϏε ElasticsearchΛΤϯδϯ ͱͨ͠ϚωʔδυαʔϏε AWSͷ؂ࢹπʔϧ

50. Ϋϥ΢υͰࣄނΛ๷͙ͨΊʹ

51. ઈର΍Δ΂͖͜ͱ • ྉۚΞϥʔϜΛઃఆ͢Δ
 Ξϥʔτͱ௨஌Ͱ੥ٻֹΛϞχλϦϯά
 http://docs.aws.amazon.com/ja_jp/awsaccountbilling/latest/aboutv2/monitor-charges.html • APIΩʔͷ؅ཧ͸ެ։ઈର͠ͳ͍ • Access key/Secret

    Key͸ඞͣެ։͠ͳ͍Α͏ʹ • ʮॳ৺ऀ͕AWSͰϛεͬͯෆਖ਼ར༻͞Εͯ$6,000੥ٻɺٽ͖ͦ͏ʹͳ͓ͬͨ࿩ʯͳͲ • ݟੵ΋Γ͸ඞ͓ͣ͜ͳ͏ • AWS ؆қݟੵΓπʔϧ
 http://calculator.s3.amazonaws.com/index.html?lng=ja_JP • AWS ՝ۚମܥͱݟੵΓํ๏ʹ͍ͭͯ
 https://aws.amazon.com/jp/how-to-understand-pricing/ • ηΩϡϦςΟάϧʔϓ͸[ssh 0.0.0.0/0]ͱ͔ઃఆ͠ͳ͍ • ݁ߏϒϧʔτϑΥʔε(૯౰Γ߈ܸ)ͱ͔ɺ੬ऑੑ߈ܸ͕͋Γ·͢

52. ࣭໰λΠϜ

53. AWS΍σʔλαΠΤϯε ΍Γ͍ͨͻͱ ϚδͰઈࢍืूத ※ΦϑΟεݟֶ͚ͩͰ΋େৎ෉Ͱ͢ αΠόʔΤʔδΣϯτ ΞυςΫελδΦ https://adtech.cyberagent.io/

54. ͋Γ͕ͱ͏͍͟͝·ͨ͠

55. ࢀߟ৘ใ AWS υΩϡϝϯτ https://aws.amazon.com/jp/documentation/ AWS Ϋϥ΢υαʔϏε׆༻ࢿྉू https://aws.amazon.com/jp/aws-jp-introduction/ Youtube AWSެࣜνϟϯωϧ(Re:InventͷηογϣϯͳͲ) ӳޠ

    https://www.youtube.com/user/AmazonWebServices ೔ຊޠ https://www.youtube.com/user/AmazonWebServicesJP 
AWSΫϥ΢υσβΠϯύλʔϯ http://aws.clouddesignpattern.org/index.phpύλʔϯ Developers.IO http://dev.classmethod.jp/ DeNA TechCon 2017 ͱ Developers Summit 2017 ͰDeNAͷػցֶशج൫ͱ෼ੳج൫ͷߨԋΛ͠·ͨ͠ http://blog.livedoor.jp/sonots/archives/49502478.html