DeNA AIシステム部におけるクラウドを活用した機械学習基盤の構築

Transcript

1. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   DeNA AIγεςϜ෦ʹ͓
   ͚ΔΫϥ΢υΛ׆༻ͨ͠
   ػցֶशج൫ͷߏங
   #denatechcon 2017
   1
   Feb 10, 2017
   Naotoshi Seo
   AI System Division

   DeNA Co., Ltd.
   

   View Slide

2. ࣗݾ঺հ
   2
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   ɾ੉ඌ௚ར!TPOPUT
   ɾ"*γεςϜ෦ چ෼ੳج൫෦
   
   ɾ"OBMZUJDT5PPMTνʔϜ
   $MPVE*OGSBνʔϜ
   ϦʔυΤϯδχΞ ݉೚
   
   ɾ'MVFOUE3VCZίϛολ
   

   View Slide

3. ର৅ऀ
   3
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   • ػցֶशΘ͔Βͳ͍͚Ͳػցֶशج൫Λ࡞Β
   ͳ͚Ε͹͍͚ͳ͘ͳͬͨΠϯϑϥ୲౰ऀ
   • ΠϯϑϥΘ͔Βͳ͍͚Ͳػցֶशج൫Λ࡞Β
   ͳ͚Ε͹͍͚ͳ͘ͳͬͨػցֶशΤϯδχΞ
   

   View Slide

4. ໨࣍
   4
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   • ػցֶशج൫ʹٻΊΒΕΔ ͨ
   ΋ͷ
   • ५୔ͳ(16
   • ִ཭͞Εͨ؀ڥ
   • ૉૣ͍ߏங
   • ӡ༻ָ͕
   • ࣗ༝౓Λߴ͘
   • ϛε͕͓͖ʹ͍͘
   • (16αʔόͷΦϯσϚϯυΦʔτεέʔϧ
   • "1*αʔό αʔϏεఏڙ؀ڥ
   
   • 'VUVSF8PSL
   • ͓·͚"84ͱ($1ͷൺֱ
   

   View Slide

5. 5
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   ५୔ͳ(16
   ५୔ͳGPU
   

   View Slide

6. 6
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   ͷલʹ
   

   View Slide

7. 7
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   (16ͷجૅ஌ࣝ
   GPUͷجૅ஌ࣝ
   

   View Slide

8. 8
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   • σΟʔϓϥʔχϯά
   • (16Λ࢖ͬͨ΄͏͕଎͍
   • ྫ
   $16ͩͱ೔ˠ(16ͩͱ೔
   ͳͥ(16͕ඞཁ͔
   GPUͷجૅ஌ࣝ
   

   View Slide

9. (16ͷಛ௃
   9
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   • ฒྻॲཧ͕ಘҙ
   • $16ͩͱίΞͳͲͷΦʔμʔ
   • (16ͩͱ d ίΞ
   • ෼ذॲཧ͸ۤख
   • ෼ذ༧ଌɾΞ΢τΦϒΦʔμʔͷ
   ػೳΛ؆ૉԽ͢Δ͜ͱͰίΞ਺Λ
   ૿΍͍ͯ͠Δ
   • ߦྻԋࢉʹ޲͍͍ͯΔ
   GPUͷجૅ஌ࣝ
   

   View Slide

10. (16੡඼
    10
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • /7*%*"5FTMB
    • /7*%*"ͷ)1$޲͚(16γϦʔζ
    • /7*%*"(F'PSDF(3*%
    • Ϋϥ΢υήʔϛϯά޲͚
    • ".%'JSF1SP
    • ".%ͷ)1$޲͚γϦʔζ
    GPUͷجૅ஌ࣝ
    

    View Slide

11. /7*%*"5FTMB
    11
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ੈ୅
    • 'FSNJ,FQMFS
    • .BYXFMM1BTDBM
    • ྫ
    LˠLFQMFS
    • "1*
    • $6%" /7*%*"ಠࣗ
    
    • 0QFO$-
    • %JSFDU$PNQVUF
    GPUͷجૅ஌ࣝ
    http://pc.watch.impress.co.jp/img/pcw/docs/755/994/html/Visio-
    NvidiaGPUComparisonGFLOPSvsMemorybandwidth.png.html
    

    View Slide

12. $6%"ͷΞʔΩςΫνϟ
    12
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ϗετ $16
    ͔ΒσόΠ
    ε (16
    ʹσʔλΛసૹ
    • (16Ͱॲཧ
    • ݁ՌΛσόΠε (16
    ͔
    Βϗετʹసૹ $16
    
    GPUͷجૅ஌ࣝ
    http://www.nvidia.co.jp/docs/IO/51174/
    NVIDIA_CUDA_Programming_Guide_1.1_JPN.pdf
    

    View Slide

13. 13
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ५୔ͳ(16
    ५୔ͳGPU
    

    View Slide

14. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    DeNAͷΦϯϓϨGPU؀ڥ
    (2017/02ݱࡏ)
    14
    CPUɿXeon E5-2630 v3 2.4GHz *2
    GPUɿNVIDIA Tesla M40 *2
    Memɿ64GB
    NICɿ10Gb ୆
    ίΞ਺ MHz GPUϝϞϦ ൃද೥
    M40 3072 1114 12GB 2015/11
    ५୔ͳGPU
    

    View Slide

15. 15
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ଍Γͳ͍
    ɾТɾʆ
    
    ५୔ͳGPU
    

    View Slide

16. "*Ҋ݅ʹ͓͚Δ(16ར༻
    16
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ͋Ε͹͋Δ͚ͩ࢖͍͍ͨ
    • ύϥϝʔλΛগ͠ม͑ͯฒྻͰֶशͤ͞
    ͯɺྑ͍ํΛબͿ
    • ֶश͢Δ࣌ʹ͚ͩ࢖͏
    • ऴΘͬͨΒഇغͯ͠΋ߏΘͳ͍
    ५୔ͳGPU
    

    View Slide

17. 17
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ͦ͏ͩ
    Ϋϥ΢υͩ
    ५୔ͳGPU
    

    View Slide

18. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ५୔ͳGPU
    DHΠϯελϯε
    HΠϯελϯε
    QΠϯελϯε
    چੈ୅
    AWSͷGPU
    

    View Slide

19. HΠϯελϯε
    19
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    AWSͷGPU
    GPU ίΞ਺ MHz ൃද೥
    NVIDIA GRID K520 1536 * 2 800 2013/5
    Πϯελϯε໊ GPU

    ݸ
    vCPU
    ݸ
    ϝϞϦ
    (GiB)
    ฒྻί
    Ξ
    GPUϝ
    ϞϦ
    (GB)
    ωο
    τϫ
    ʔΫ
    Ձ֨/࣌
    (us-
    east-1)
    g2.2xlarge 1 8 15 1,536 4 $0.65
    g2.8xlarge 4 32 60 6,144 32 $2.6
    

    View Slide

20. QΠϯελϯε
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ౦ژϦʔδϣϯͰ(·ͩ)࢖͑ͳ͍
    AWSͷGPU
    GPU ίΞ਺ MHz ൃද೥
    NVIDIA Tesla K80 2496 * 2 875 2014/11
    Πϯελϯε໊ GPU
    ݸ
    vCPU
    ݸ
    ϝϞ
    Ϧ
    (GiB)
    ฒྻί
    Ξ
    GPUϝ
    ϞϦ
    (GB)
    ωοτ
    ϫʔΫ
    Ձ֨/࣌
    (us-
    east-1)
    p2.xlarge 1 4 61 2,496 12 ߴ $0.9
    p2.8xlarge 8 32 488 19,968 96 10G $7.2
    p2.16xlarge 16 64 732 39,936 192 20G $14.4
    

    View Slide

21. %F/"Ͱͷ(16ར༻
    21
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • VTͰQΠϯελϯε L
    Λ࢖͏
    • ֶशͤ͞Δ͚࣌ͩىಈ͠ɺ͋ͱ͸ࢭΊΔ
    • ϋΠύʔύϥϝʔλνϡʔχϯάͳͲͰͨ
    ͘͞Μฒ΂࣮ͯߦ͢ΔͷʹΫϥ΢υ؀ڥ͸
    ༗ར‷ ŋŷŋ
    و ̑̑
    ५୔ͳGPU
    

    View Slide

22. ($1ͷ(16
    22
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    GCPͷGPU
    https://cloud.google.com/gpu/
    

    View Slide

23. ($1ͷ(16
    23
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ΞϧϑΝ
    • ਃ੥͢Ε͹L QΠ
    ϯελϯεͱಉ͡
    ͕࢖
    ͑Δ
    • ͦΕҎ֎͸·ͩμϝͩͬ
    ͨ ɾТɾʆ
    
    • L͕௨ৗΠϯελϯ
    εʹΞλον͞Εͨܗ
    GCPͷGPU
    

    View Slide

24. ($1ͷ(16
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ௨ৗΠϯελϯε୅ۚʹΞλον෼͚ͩϓϥε
    GCP͸Πϯελϯεͷ՝͕ۚ෼୯Ґ(ͨͩ͠ɺ
    ࠷௿Ͱ΋10෼͸औΒΕΔ)
    GCPͷGPU
    GPU
    ݸ
    ฒྻί
    Ξ
    GPUϝ
    ϞϦ
    (GB)
    FP32

    (TFLOP
    S)
    Ձ֨/࣌
    (us)
    Ϧʔδϣ
    ϯ
    Tesla K80x1 1 2,496 12 4.3 $0.7 ౦ژҎ֎
    Tesla P100x1 1 3,584 16 10.6 ෆ໌ N/A
    AMD FirePro
    S9300 x2
    1 8 13.9 ෆ໌ N/A
    

    View Slide

25. ػցֶशج൫ʹٻΊΒΕΔ ͨ
    ΋ͷ
    25
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ɾ५୔ͳ(16
    ɾ
    ִ཭͞Εͨ؀ڥ
    ɾૉૣ͍ߏங
    ɾӡ༻ָ͕
    ɾࣗ༝౓Λߴ͘
    ɾϛε͕͓͖ʹ͍͘
    

    View Slide

26. ִ཭͞Εͨ؀ڥ
    26
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • %F/"ͷ"*Ҋ݅
    
    • ൿಗҊ݅
    • ڠۀҊ݅
    • ඇؔ༩ࣾһ͕σʔλʹΞΫηεͰ͖ͳ͍Α͏ʹִ
    ཭͞ΕͨηΩϡΞͳ؀ڥΛ࡞Δඞཁ͕͋Δ
    • ͨͩ͠ɺڠۀձࣾͷؔ༩ࣾһ͸ΞΫηεͰ͖Δ
    Ϋϥ΢υΞΧ΢ϯτΛ෼͚ͯ
    ผݸͷ؀ڥͱִͯ͠཭
    ִ཭͞Εͨ؀ڥ
    

    View Slide

27. ωοτϫʔΫߏ੒
    27
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Subnet
    service
    components
    step
    DeNA
    PC
    Gateway
    ghe, etc
    Security Group
    ssh
    Security Group
    ִ཭͞Εͨ؀ڥ
    • ΦϯϓϨ(BUFXBZ
    ͔ΒͷΈ౿Έ୆αʔ
    όʹTTIՄೳ
    • (BUFXBZͰάϧʔ
    ϓݖݶ؅ཧ
    • ίʔυ͸HJUIVCF
    ʹઐ༻PSHΛ࡞ͬ
    ͯQSJWBUFӡ༻
    

    View Slide

28. ؂ࠪϩά
    28
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ίϚϯυ࣮ߦϩά
    • %F/"ͷ(BUFXBZʹ͋ΔΩʔϩΨʔ
    • Ϋϥ΢υͷૢ࡞ϩά
    • "84$MPVE5SBJM
    • ($14UBDL%SJWFS
    ִ཭͞Εͨ؀ڥ
    

    View Slide

29. ػցֶशج൫ʹٻΊΒΕΔ ͨ
    ΋ͷ
    29
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ɾ५୔ͳ(16
    ɾִ཭͞Εͨ؀ڥ
    ɾ
    ૉૣ͍ߏங
    ɾӡ༻ָ͕
    ɾࣗ༝౓Λߴ͘
    ɾϛε͕͓͖ʹ͍͘
    

    View Slide

30. ૉૣ͍ߏங
    30
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • "*Ҋ͕݅ͲΜͲΜ্ཱ͕ͪͬͯ͘Δ
    • ̍ϲ݄ʹ̑ϓϩδΣΫτ ݄
    
    • ઐ೚͡Όͳ͍ͷͰɺखؒͳ͘ૉૣ͘
    ૉૣ͍ߏங
    

    View Slide

31. ࣗಈԽίʔυԽ
    31
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • 5FSSBGPSNͰΦʔέετϨʔγϣϯͷ
    ࣗಈԽˍίʔυԽ
    • "84 ($1྆ํʹରԠ
    • *UBNBFͰϓϩϏδϣχϯά
    • ੿࡞ͷLPOEBUFϥούʔ
    ૉૣ͍ߏங
    

    View Slide

32. ػցֶश؀ڥͷ
    ϓϩϏδϣχϯά
    32
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • VCVOUV$IBJOFSΦεεϝ
    
    • OWJEJBESJWFS
    • DVEBOWJEJBͷDVEBBQUSFQP͔ΒೖΕΔͱOWJEJBESJWFS΋ೖָͬͯ
    • DVEOOOWJEJBͷNBDIJOFMFBSOJOHBQUSFQP͔ΒೖΕΔͱָ
    
    • QZUIPO
    • DIBJOFS
    ૉૣ͍ߏங
    

    View Slide

33. ػցֶशج൫ʹٻΊΒΕΔ ͨ
    ΋ͷ
    33
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ɾ५୔ͳ(16
    ɾִ཭͞Εͨ؀ڥ
    ɾૉૣ͍ߏங
    
    ɾ
    ӡ༻ָ͕
    ɾࣗ༝౓Λߴ͘
    ɾϛε͕͓͖ʹ͍͘
    

    View Slide

34. ӡ༻ָ͕
    34
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • "*Ҋ͕݅ͲΜͲΜ্ཱ͕ͪͬͯ͘Δ
    • ͲΜͲΜ؀ڥ͕૿͑Δ
    • ઐ೚͡Όͳ͍ͷͰɺָʹ͠ͳ͍ͱ݉ۀͷ
    ΞϓϦ։ൃۀ͕Ͱ͖ͳ͍
    ӡ༻ָ͕
    

    View Slide

35. 35
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ͦ͏ͩ
    Ϋϥ΢υͩ
    ५୔ͳGPU
    

    View Slide

36. Ϋϥ΢υͷػೳΛ࢖͏
    36
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • Ͱ͖Δ͚ͩΫϥ΢υͷػೳΛ࢖͏
    • ؂ࢹπʔϧ
    • "84$MPVE8BUDI
    • ($14UBDL%SJWFS
    • .Z42- "843%4 ($1$MPVE42-
    
    • 3FEJT "84&MBTUJDBDIF
    ͳͲͳͲ
    ӡ༻ָ͕
    

    View Slide

37. ར༻தͷ"84αʔϏε
    37
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • &$
    • 71$
    •
    • 3PVUF
    • 4
    • "VSPSBPS3%4 .Z42-
    
    • &MBTUJ$BDIF 3FEJT
    
    • &MBTUJD-PBE#BMBODJOH
    • $MPVE'SPOU
    • &MBTUJD'JMF4ZTUFN /'4
    
    • 4/4
    • 4&4 4.51
    
    • &.3 )BEPPQ 4QBSL
    
    • $MPVE8BUDI
    • $MPVE5SBJM
    ӡ༻ָ͕
    

    View Slide

38. ར༻தͷ($1αʔϏε
    38
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • (PPHMF$PNQVUF&OHJOF ($&
    
    • (PPHMF"QQ&OHJOF
    • (PPHMF-PBE#BMBODJOH
    • (PPHMF$MPVE$%/
    • (PPHMF$MPVE%/4
    • (PPHMF$MPVE42-
    • 4UBDLESJWFS.POJUPSJOH
    • (PPHMF$MPVE4UPSBHF
    • (PPHMF#JH2VFSZ
    • (PPHMF$MPVE%BUBQSPD
    ӡ༻ָ͕
    

    View Slide

39. ػցֶशج൫ʹٻΊΒΕΔ ͨ
    ΋ͷ
    39
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ɾ५୔ͳ(16
    ɾִ཭͞Εͨ؀ڥ
    ɾૉૣ͍ߏங
    ɾӡ༻ָ͕
    ɾ
    ࣗ༝౓Λߴ͘
    ɾϛε͕͓͖ʹ͍͘
    

    View Slide

40. ࣗ༝౓Λߴ͘
    40
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • γε ΤϯδχΞ
    ʹ΋̎ύλʔϯ
    Πϯϑϥ͍͡Γ͍ͨ
    Πϯϑϥ͓೚ͤ͠·͢
    • ͍͡Γ͍ͨͳΒੋඇ΍ͬͯ͘Εʂ
    ࣗ༝౓Λߴ͘
    

    View Slide

41. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    41
    https://suzuri.jp/mizzy/76284/t-shirt/m/navy
    

    View Slide

42. ͍ͬͯ͡΋ྑ͍
    42
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • Πϯελϯεͷ্ཱͪ͛͸ࣗ༝
    • Ϋϥ΢υαʔϏεͷઃఆ
    • 5FSSBGPSNʹ13
    • ϓϩϏδϣχϯά
    • ج൫ͱ͸ผͷπʔϧͰίʔυԽ΋0,
    • खಈηοτΞοϓͯ͠ΠϝʔδʹݻΊ
    ͯ΋0,
    ࣗ༝౓Λߴ͘
    

    View Slide

43. ػցֶशج൫ʹٻΊΒΕΔ ͨ
    ΋ͷ
    43
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ɾ५୔ͳ(16
    ɾִ཭͞Εͨ؀ڥ
    ɾૉૣ͍ߏங
    ɾӡ༻ָ͕
    ɾࣗ༝౓Λߴ͘
    ɾ
    ϛε͕͓͖ʹ͍͘
    

    View Slide

44. ϛε͕͓͖ʹ͍͘
    44
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ࣗ༝౓Λߴ͘͢Δͱࠓ౓͸ϛε͕͓͖Δ
    • ͨͱ͑͹ɺΠϯελϯε࡞੒
    • ద੾ͳηΩϡϦςΟάϧʔϓΛ෇͚Δ
    • λάΛ෇͚Δ
    • 3PVUFʹϗετ໊ొ࿥
    • ؂ࢹ౤ೖ
    ࣗ࡞πʔϧͰ୲อ
    ϛε͕͓͖ʹ͍͘
    

    View Slide

45. FDDSFBUFJOTUBODFT
    HDFDSFBUFJOTUBODFT
    
    45
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ΠϯελϯεΛ࡞੒
    • 3PMFTλάΛ෇͚Δ
    • ద੾ͳΠϝʔδΛબ୒ ࣍΃ʔδ
    
    • ద੾ͳηΩϡϦςΟάϧʔϓΛ෇͚Δ
    • 3PVUFʹϗετ໊ొ࿥
    • ($1ͷ৔߹͸ඪ४Ͱ%/4ొ࿥͞ΕΔͷͰෆཁ
    • ؂ࢹ౤ೖ
    $ ec2-create-instances -n host --role chainer:dev
    ϛε͕͓͖ʹ͍͘
    

    View Slide

46. FDDSFBUFBNJ
    HDFDSFBUFTOBQTIPU
    
    46
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ࢦఆͨ͠ϗετΛΠϝʔδԽ
    • Πϝʔδʹ΋ϗετͱಉ͡ϩʔϧΛ෇͚Δ
    • ࣍ճFDDSFBUFJOTUBODFT͢Δࡍʹ͸ɺ
    ಉ͡ϩʔϧͷ".*ͷ͏ͪɺ࠷৽ͷ΋ͷ͔ΒΠ
    ϯελϯεΛཱͪ͋͛Δ
    $ ec2-create-ami -n host
    ϛε͕͓͖ʹ͍͘
    

    View Slide

47. FDMJTUBNJ
    HDFMJTUTOBQTIPUT
    
    47
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ࢦఆͨ͠ϩʔϧͷ͍ͭͨ".*ͷ͏ͪɺ࠷৽ͷ
    ΋ͷΛදࣔ͢Δ
    • FDDSFBUFJOTUBODFT͸಺෦తʹ͜Ε
    Λ ϥΠϒϥϦͱͯ͠
    ར༻ͯ͠ɺద੾ͳ".*
    Λબ୒͢Δ
    $ ec2-list-ami --role chainer:dev
    ami-XXXXXXXX 2017-02-04T03:53:10+09:00 chainer:dev
    ϛε͕͓͖ʹ͍͘
    

    View Slide

48. FDIPTU HDFIPTU
    
    48
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • &$Πϯελϯεʹ
    3PMFTλάΛ෇͚ͯɺϩʔ
    ϧͰ؅ཧɺݕࡧ
    • ($&ͷ৔߹͸
    NFUBEBUBΛར༻
    
    • *UBNBFͷϩʔϧࢦఆ࣌
    ʹ΋ར༻
    $ ec2-host --role chainer:dev
    asaa5002
    ϛε͕͓͖ʹ͍͘
    

    View Slide

49. ໨࣍
    49
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ɾػցֶशج൫ʹٻΊΒΕΔ ͨ
    ΋ͷ
    ɾ(16αʔόͷΦϯσϚϯυΦʔτεέʔϧ
    ɾ"1*αʔό αʔϏεఏڙ؀ڥ
    
    ɾ'VUVSF8PSL
    ɾ͓·͚"84ͱ($1ͷൺֱ
    

    View Slide

50. (16αʔόͷ
    ΦϯσϚϯυΦʔτεέʔϧ
    50
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ฒྻͰֶशΛ૸Β͍ͤͨ
    • (16ΠϯελϯεΛඞཁͳ෼͚ͩࣗಈ
    Ͱ্ཱͪ͛ͯཉ͍͠
    • ֶश͕ऴΘͬͨΒࣗಈͰམͱͯ͠΄͍͠
    • མͱ͠๨Ε͕ͪʼʻ
    GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ
    

    View Slide

51. ...
    Chainer Cluster
    create
    ssh
    terminate
    ...
    XXX Cluster
    FDTDBMFSVO
    51
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ֶशͤ͞Δͱ͖ʹΠϯελϯεΛ
    ্ཱͪ͛ͯίϚϯυ࣮ߦ
    • ͢Ͱʹ্ཱ͕͍ͪͬͯͯ࢖ΘΕͯ
    ͍ͳ͍Πϯελϯε͕͍Ε͹ੵۃ
    తʹར༻͢Δ
    • "84͸̍࣌ؒ՝ۚͳͷͰ͙͢ʹ͸
    ࡴ͞ͳ͍
    • ىಈ࣌ؒNJONJO
    Ͱ࢖ΘΕ͍ͯͳ͍Πϯελϯε͕
    ͍Ε͹མͱ͢
    GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ
    $ ec2-scale-run -c config.yml [ |
    

    View Slide

52. 52
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    མͱ͠๨Εͳ͠
    ‷ ŋŷŋ
    و ̑̑
    

    View Slide

53. %PDLFSରԠ
    53
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    TDBMFSVOͷதͰOWJEJBEPDLFSSVO
    ྫ
    $IBJOFS5FOTPS'MPXίϯςφ
    "84($1Ͱखݩͱಉ͡ίϯςφΛ࢖͑ͯศར
    2&$4͸࢖Θͳ͍ͷʁ
    "ࠓͷͱ͜Ζ࢖ͬͯ·ͤΜ
    ཧ༝(16Λ࢖͑ΔΑ͏ʹ͢Δઃఆ͕
    ৭ʑΊΜͲ͏ͩͬͨ
    ཧ༝(,&ͱͷΠϯλʔϑΣʔεҧ͍
    GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ
    

    View Slide

54. Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    https://suzuri.jp/chiastolite/90304/t-shirt/s/royalblue
    GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ
    

    View Slide

55. ิ଍OWJEJBEPDLFS
    55
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • EPDLFSDMJͷബ͍ϥούʔ
    • %PDLFSίϯςφ͔Βϗετͷ(16
    Λ࢖͏ͱ͖ʹศར
    • OWJEJBEPDLFSQMVHJO͔Β৘
    ใΛಘͯɺEPDLFSίϚϯυʹΦ
    ϓγϣϯΛ৭ʑ౉ͯ͠΋ྑ͍͕໘౗
    $ nvidia-docker run nvidia/cuda nvidia-smi
    GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ
    

    View Slide

56. /'4Ϛ΢ϯτ
    56
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ΠϯελϯεΛ্ཱͪ͛
    ͯ͸མͱ͢ͷͰɺ
    • ڞ༗ετϨʔδ͕ඞཁ
    • ֶशݩσʔλ
    • தؒεφοϓγϣοτ
    NFS
    • "NB[PO&'4ΛϚ΢ϯτ
    • ༰ྔࣗಈ֦ு
    • ஋ஈ͸͓ߴΊ (#݄
    
    • ౦ژϦʔδϣϯͰ͸࢖͑ͳ͍
    GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ
    

    View Slide

57. εϙοτΠϯελϯεͷ׆༻
    57
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ೖࡳՁ֨ࢢ৔ εϙοτ
    Ձ֨ͳΒɺࢢ৔Ձ֨
    ͰΠϯελϯε্ཱ͕͕ͪΔ
    • ࢢ৔Ձ͕͕֨͋ͬͨΒɺՁ͕֨มΘΔ
    • ࢢ৔Ձ֨ೖࡳՁ֨ʹٯసͨ͠ΒɺΠϯελϯ
    ε͕མͱ͞ΕΔ
    εϙοτΠϯελϯε
    
    QYMBSHFIˠI 0''
    
    GPUαʔόͷΦϯσϚϯυΦʔτεέʔϧ
    

    View Slide

58. 58
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    "1*αʔό
    αʔϏεఏڙ؀ڥ
    
    

    View Slide

59. "1*αʔό
    59
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ػցֶश̎ͭͷϑΣʔζ
    • ֶशϑΣʔζ
    • ༧ଌϑΣʔζ
    • ֶशϑΣʔζ͸։ൃ؀ڥͰ (16ར༻
    
    • ༧ଌϑΣʔζ͸ϓϩμΫγϣϯ $16ར༻
    
    • )551"1*ͱͯͯ͠͠αʔϏεʹఏڙ
    APIαʔό
    

    View Slide

60. ࣮ྫ
    60
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Elastic Load Balancing
    ...
    API Server (Blue)
    ...
    API Server (Green)
    • $IBJOFSͷ༧ଌϑΣʔ
    ζΛ࢖͏ͷͰ1ZUIPO
    ΞϓϦέʔγϣϯ
    • %KBOHP
    • ϩʔΧϧ+VCBUVT
    • #MVF(SFFO
    %FQMPZNFOU
    APIαʔό
    ap-northeast-1 (cpu)
    

    View Slide

61. 61
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    'VUVSF8PSL
    

    View Slide

62. 'VUVSF8PSL
    62
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • "1*αʔόΛ("& (PPMHF"QQ&OHJOF
    
    Ͱӡ༻ָ͍ͯͨ͠͠
    • ("&QZUIPO
    • 4UBOEBSE&OWJSPONFOU͸੍໿͕ݫ͠
    ͗͢ΔͷͰ'MFYJCMF&OWJSPONFOU͔
    • શࣗಈϋΠύʔύϥϝʔλνϡʔχϯά͞Μ
    Future Work
    

    View Slide

63. 'VUVSF8PSL
    63
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ෺ཧ(16Ϛγϯ΋ཉ͍͠ ɾТɾʆ
    
    • τϥΠΞϯυΤϥʔͳ։ൃ༻్
    • खݩͰ։ൃ͍ͨ͠
    • େྔʹฒྻͰֶश͍ͤͨ͞৔߹͸Ϋϥ΢υ
    • ࠷௿Ͱ΋̍ਓ̍୆ߪೖ༧ఆ‷ ŋŷŋ
    و ̑̑
    Future Work
    

    View Slide

64. 64
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    ͓·͚
    

    View Slide

65. 65
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    "84ͱ($1
    ͷൺֱ
    

    View Slide

66. Πϯελϯεྉۚ
    66
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • (16Πϯελϯεͷ஋ஈ͸͋·ΓมΘΒͳ͍
    • ($&Πϯελϯεͷ՝ۚ͸̍෼͝ͱ
    • ͨͩ͠ɺ࠷௿Ͱ΋෼͸औΒΕΔ
    • ($&͸Πϯελϯε͕̍ϲ݄࣮ؒߦ͞ΕΔ
    ͱɺྉ͕ۚࣗಈͰׂΓҾ͔ΕΔ
    • &$ͷϦβʔϒυΠϯελϯεͱҧ͍ɺ৽͍͠Π
    ϯελϯελΠϓ͕ग़ͨ࣌ʹҾͬӽ͠΍͍͢ҹ৅
    AWSͱGCPͷൺֱ
    

    View Slide

67. ($1ͷϩʔυόϥϯα͕ڧ͍
    67
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ΢ΥʔϜΞοϓෆཁ
    • ࣄલεέʔϧΞ΢τਃ੥ෆཁ
    • ϚϧνϦʔδϣϯରԠ
    • ̍άϩʔόϧ*1 *1ΤχʔΩϟετ
    
    AWSͱGCPͷൺֱ
    

    View Slide

68. ($1ͷωοτϫʔΫઃఆ͕ॊೈ
    68
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ΠϯελϯεʹλάΛ෇͚ͯɺλάͰϧʔ
    ςΟϯάઃఆՄೳ
    • "84ͷ৔߹͸TVCOFUͰϧʔςΟϯάઃఆ
    AWSͱGCPͷൺֱ
    

    View Slide

69. "84ͷ΄͏͕αʔϏε͸ॆ࣮
    69
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ($1ʹ͸"84&MBTUJ$BDIF 3FEJT
    .FNDBDIFE
    ૬౰͕ͳ͍
    • &'4 /'4
    ૬౰͕ͳ͍
    • 4.51αʔϏε͕ͳ͍
    • SEQBSUZͱఏܞ
    • ଍Γͳ͍ͱ͜Ζ͕ͪΒ΄Β
    • 4UBDL%SJWFSʹ؂ࢹొ࿥"1*͕·ͩͳ͍
    AWSͱGCPͷൺֱ
    

    View Slide

70. ·ͱΊ
    70
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ػցֶशج൫ʹٻΊΒΕΔ ͨ
    ΋ͷ
    • ५୔ͳ(16
    • ִ཭͞Εͨ؀ڥ
    • ૉૣ͍ߏங
    • ӡ༻ָ͕
    • ࣗ༝౓Λߴ͘
    • ϛε͕͓͖ʹ͍͘
    • (16αʔόͷΦϯσϚϯυΦʔτεέʔϧ
    • "1*αʔό αʔϏεఏڙ؀ڥ
    
    • 'VUVSF8PSL ("& શࣗಈύϥϝʔλνϡʔχϯά
    
    ˠͦΕΫϥ΢υͰ
    

    View Slide

71. 4QFDJBM5IBOLT
    ॱෆಉ
    
    71
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • OIBZBTIJ
    • LB[VLJGVKJLBXB
    • GVLVEBNB 1'/
    
    • DJWJUBTQP
    • NB[HJ
    

    View Slide