DeNA AIシステム部におけるクラウドを活用した機械学習基盤の構築

by Naotoshi Seo

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

/7*%*"5FTMB 11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ੈ୅ • 'FSNJ,FQMFS • .BYXFMM1BTDBM • ྫ LˠLFQMFS • "1* • $6%" /7*%*"ಠࣗ • 0QFO$- • %JSFDU$PNQVUF GPUͷجૅ஌ࣝ http://pc.watch.impress.co.jp/img/pcw/docs/755/994/html/Visio- NvidiaGPUComparisonGFLOPSvsMemorybandwidth.png.html

Slide 12

Slide 12 text

$6%"ͷΞʔΩςΫνϟ 12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ϗετ $16 ͔ΒσόΠ ε (16 ʹσʔλΛసૹ • (16Ͱॲཧ • ݁ՌΛσόΠε (16 ͔ Βϗετʹసૹ $16 GPUͷجૅ஌ࣝ http://www.nvidia.co.jp/docs/IO/51174/ NVIDIA_CUDA_Programming_Guide_1.1_JPN.pdf

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DeNAͷΦϯϓϨGPU؀ڥ (2017/02ݱࡏ) 14 CPUɿXeon E5-2630 v3 2.4GHz *2 GPUɿNVIDIA Tesla M40 *2 Memɿ64GB NICɿ10Gb ୆ ίΞ਺ MHz GPUϝϞϦ ൃද೥ M40 3072 1114 12GB 2015/11 ५୔ͳGPU

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

HΠϯελϯε 19 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. AWSͷGPU GPU ίΞ਺ MHz ൃද೥ NVIDIA GRID K520 1536 * 2 800 2013/5 Πϯελϯε໊ GPU  ݸ vCPU ݸ ϝϞϦ (GiB) ฒྻί Ξ GPUϝ ϞϦ (GB) ωο τϫ ʔΫ Ձ֨/࣌ (us- east-1) g2.2xlarge 1 8 15 1,536 4 $0.65 g2.8xlarge 4 32 60 6,144 32 $2.6

Slide 20

Slide 20 text

QΠϯελϯε Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ౦ژϦʔδϣϯͰ(·ͩ)࢖͑ͳ͍ AWSͷGPU GPU ίΞ਺ MHz ൃද೥ NVIDIA Tesla K80 2496 * 2 875 2014/11 Πϯελϯε໊ GPU ݸ vCPU ݸ ϝϞ Ϧ (GiB) ฒྻί Ξ GPUϝ ϞϦ (GB) ωοτ ϫʔΫ Ձ֨/࣌ (us- east-1) p2.xlarge 1 4 61 2,496 12 ߴ $0.9 p2.8xlarge 8 32 488 19,968 96 10G $7.2 p2.16xlarge 16 64 732 39,936 192 20G $14.4

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

($1ͷ(16 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ௨ৗΠϯελϯε୅ۚʹΞλον෼͚ͩϓϥε GCP͸Πϯελϯεͷ՝͕ۚ෼୯Ґ(ͨͩ͠ɺ ࠷௿Ͱ΋10෼͸औΒΕΔ) GCPͷGPU GPU ݸ ฒྻί Ξ GPUϝ ϞϦ (GB) FP32  (TFLOP S) Ձ֨/࣌ (us) Ϧʔδϣ ϯ Tesla K80x1 1 2,496 12 4.3 $0.7 ౦ژҎ֎ Tesla P100x1 1 3,584 16 10.6 ෆ໌ N/A AMD FirePro S9300 x2 1 8 13.9 ෆ໌ N/A

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

ωοτϫʔΫߏ੒ 27 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Subnet service components step DeNA PC Gateway ghe, etc Security Group ssh Security Group ִ཭͞Εͨ؀ڥ • ΦϯϓϨ(BUFXBZ ͔ΒͷΈ౿Έ୆αʔ όʹTTIՄೳ • (BUFXBZͰάϧʔ ϓݖݶ؅ཧ • ίʔυ͸HJUIVCF ʹઐ༻PSHΛ࡞ͬ ͯQSJWBUFӡ༻

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

ػցֶश؀ڥͷ ϓϩϏδϣχϯά 32 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • VCVOUV$IBJOFSΦεεϝ • OWJEJBESJWFS • DVEBOWJEJBͷDVEBBQUSFQP͔ΒೖΕΔͱOWJEJBESJWFS΋ೖָͬͯ • DVEOOOWJEJBͷNBDIJOFMFBSOJOHBQUSFQP͔ΒೖΕΔͱָ • QZUIPO • DIBJOFS ૉૣ͍ߏங

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

ར༻தͷ"84αʔϏε 37 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • &$ • 71$ • • 3PVUF • 4 • "VSPSBPS3%4 .Z42- • &MBTUJ$BDIF 3FEJT • &MBTUJD-PBE#BMBODJOH • $MPVE'SPOU • &MBTUJD'JMF4ZTUFN /'4 • 4/4 • 4&4 4.51 • &.3 )BEPPQ 4QBSL • $MPVE8BUDI • $MPVE5SBJM ӡ༻ָ͕

Slide 38

Slide 38 text

ར༻தͷ($1αʔϏε 38 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • (PPHMF$PNQVUF&OHJOF ($& • (PPHMF"QQ&OHJOF • (PPHMF-PBE#BMBODJOH • (PPHMF$MPVE$%/ • (PPHMF$MPVE%/4 • (PPHMF$MPVE42- • 4UBDLESJWFS.POJUPSJOH • (PPHMF$MPVE4UPSBHF • (PPHMF#JH2VFSZ • (PPHMF$MPVE%BUBQSPD ӡ༻ָ͕

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Slide 45

Slide 45 text

FDDSFBUFJOTUBODFT HDFDSFBUFJOTUBODFT 45 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ΠϯελϯεΛ࡞੒ • 3PMFTλάΛ෇͚Δ • ద੾ͳΠϝʔδΛબ୒ ࣍΃ʔδ • ద੾ͳηΩϡϦςΟάϧʔϓΛ෇͚Δ • 3PVUFʹϗετ໊ొ࿥ • ($1ͷ৔߹͸ඪ४Ͱ%/4ొ࿥͞ΕΔͷͰෆཁ • ؂ࢹ౤ೖ $ ec2-create-instances -n host --role chainer:dev ϛε͕͓͖ʹ͍͘

Slide 46

Slide 46 text

FDDSFBUFBNJ HDFDSFBUFTOBQTIPU 46 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ࢦఆͨ͠ϗετΛΠϝʔδԽ • Πϝʔδʹ΋ϗετͱಉ͡ϩʔϧΛ෇͚Δ • ࣍ճFDDSFBUFJOTUBODFT͢Δࡍʹ͸ɺ ಉ͡ϩʔϧͷ".*ͷ͏ͪɺ࠷৽ͷ΋ͷ͔ΒΠ ϯελϯεΛཱͪ͋͛Δ $ ec2-create-ami -n host ϛε͕͓͖ʹ͍͘

Slide 47

Slide 47 text

FDMJTUBNJ HDFMJTUTOBQTIPUT 47 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ࢦఆͨ͠ϩʔϧͷ͍ͭͨ".*ͷ͏ͪɺ࠷৽ͷ ΋ͷΛදࣔ͢Δ • FDDSFBUFJOTUBODFT͸಺෦తʹ͜Ε Λ ϥΠϒϥϦͱͯ͠ ར༻ͯ͠ɺద੾ͳ".* Λબ୒͢Δ $ ec2-list-ami --role chainer:dev ami-XXXXXXXX 2017-02-04T03:53:10+09:00 chainer:dev ϛε͕͓͖ʹ͍͘

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

... Chainer Cluster create ssh terminate ... XXX Cluster FDTDBMFSVO 51 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ֶशͤ͞Δͱ͖ʹΠϯελϯεΛ ্ཱͪ͛ͯίϚϯυ࣮ߦ • ͢Ͱʹ্ཱ͕͍ͪͬͯͯ࢖ΘΕͯ ͍ͳ͍Πϯελϯε͕͍Ε͹ੵۃ తʹར༻͢Δ • "84͸̍࣌ؒ՝ۚͳͷͰ͙͢ʹ͸ ࡴ͞ͳ͍ • ىಈ࣌ؒNJONJO Ͱ࢖ΘΕ͍ͯͳ͍Πϯελϯε͕ ͍Ε͹མͱ͢ GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ $ ec2-scale-run -c conﬁg.yml [<‘Some shell code'> | <ﬁlename [args ...]>

Slide 52

Slide 52 text

Slide 53

Slide 53 text

%PDLFSରԠ 53 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. TDBMFSVOͷதͰOWJEJBEPDLFSSVO ྫ $IBJOFS5FOTPS'MPXίϯςφ "84($1Ͱखݩͱಉ͡ίϯςφΛ࢖͑ͯศར 2&$4͸࢖Θͳ͍ͷʁ "ࠓͷͱ͜Ζ࢖ͬͯ·ͤΜ ཧ༝(16Λ࢖͑ΔΑ͏ʹ͢Δઃఆ͕ ৭ʑΊΜͲ͏ͩͬͨ ཧ༝(,&ͱͷΠϯλʔϑΣʔεҧ͍ GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ

Slide 54

Slide 54 text

Slide 55

Slide 55 text

ิ଍OWJEJBEPDLFS 55 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • EPDLFSDMJͷബ͍ϥούʔ • %PDLFSίϯςφ͔Βϗετͷ(16 Λ࢖͏ͱ͖ʹศར • OWJEJBEPDLFSQMVHJO͔Β৘ ใΛಘͯɺEPDLFSίϚϯυʹΦ ϓγϣϯΛ৭ʑ౉ͯ͠΋ྑ͍͕໘౗ $ nvidia-docker run nvidia/cuda nvidia-smi GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ

Slide 56

Slide 56 text

Slide 57

Slide 57 text

εϙοτΠϯελϯεͷ׆༻ 57 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ೖࡳՁ֨ࢢ৔ εϙοτ Ձ֨ͳΒɺࢢ৔Ձ֨ ͰΠϯελϯε্ཱ͕͕ͪΔ • ࢢ৔Ձ͕͕֨͋ͬͨΒɺՁ͕֨มΘΔ • ࢢ৔Ձ֨ೖࡳՁ֨ʹٯసͨ͠ΒɺΠϯελϯ ε͕མͱ͞ΕΔ εϙοτΠϯελϯε <࣮੷> QYMBSHFIˠI 0'' GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Slide 60

Slide 60 text

࣮ྫ 60 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Elastic Load Balancing ... API Server (Blue) ... API Server (Green) • $IBJOFSͷ༧ଌϑΣʔ ζΛ࢖͏ͷͰ1ZUIPO ΞϓϦέʔγϣϯ • %KBOHP • ϩʔΧϧ+VCBUVT • #MVF(SFFO %FQMPZNFOU APIαʔό ap-northeast-1 (cpu)

Slide 61

Slide 61 text

Slide 62

Slide 62 text

'VUVSF8PSL 62 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • "1*αʔόΛ("& (PPMHF"QQ&OHJOF Ͱӡ༻ָ͍ͯͨ͠͠ • ("&QZUIPO • 4UBOEBSE&OWJSPONFOU͸੍໿͕ݫ͠ ͗͢ΔͷͰ'MFYJCMF&OWJSPONFOU͔ • શࣗಈϋΠύʔύϥϝʔλνϡʔχϯά͞Μ Future Work

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Πϯελϯεྉۚ 66 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • (16Πϯελϯεͷ஋ஈ͸͋·ΓมΘΒͳ͍ • ($&Πϯελϯεͷ՝ۚ͸̍෼͝ͱ • ͨͩ͠ɺ࠷௿Ͱ΋෼͸औΒΕΔ • ($&͸Πϯελϯε͕̍ϲ݄࣮ؒߦ͞ΕΔ ͱɺྉ͕ۚࣗಈͰׂΓҾ͔ΕΔ • &$ͷϦβʔϒυΠϯελϯεͱҧ͍ɺ৽͍͠Π ϯελϯελΠϓ͕ग़ͨ࣌ʹҾͬӽ͠΍͍͢ҹ৅ AWSͱGCPͷൺֱ

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Slide 69

Slide 69 text

"84ͷ΄͏͕αʔϏε͸ॆ࣮ 69 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ($1ʹ͸"84&MBTUJ$BDIF 3FEJT .FNDBDIFE ૬౰͕ͳ͍ • &'4 /'4 ૬౰͕ͳ͍ • 4.51αʔϏε͕ͳ͍ • SEQBSUZͱఏܞ • ଍Γͳ͍ͱ͜Ζ͕ͪΒ΄Β • 4UBDL%SJWFSʹ؂ࢹొ࿥"1*͕·ͩͳ͍ AWSͱGCPͷൺֱ

Slide 70

Slide 70 text

·ͱΊ 70 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ػցֶशج൫ʹٻΊΒΕΔ ͨ ΋ͷ • ५୔ͳ(16 • ִ཭͞Εͨ؀ڥ • ૉૣ͍ߏங • ӡ༻ָ͕ • ࣗ༝౓Λߴ͘ • ϛε͕͓͖ʹ͍͘ • (16αʔόͷΦϯσϚϯυΦʔτεέʔϧ • "1*αʔό αʔϏεఏڙ؀ڥ • 'VUVSF8PSL ("& શࣗಈύϥϝʔλνϡʔχϯά ˠͦΕΫϥ΢υͰ

Slide 71

Slide 71 text