Shiva 〜Nextremerをscale upする機械学習環境〜

Nextremer 社内で運用している機械学習環境 Shiva のご紹介
tag: Docker, Mesos, Marathon, Jupyter, ceph,

Transcript

1. Copyright © Nextremer, inc. All Rights Reserved. 1 Shiva ʙNextremerΛScale up͢Δػցֶश؀ڥʙ 14th Dec 2017
2. Copyright © Nextremer, inc. All Rights Reserved. 2 ࣗݾ঺հ ྆֯ ࿨ي(΋ΖͣΈ ͔͖ͣ) גࣜձࣾNextremer ϓϩάϥϚʔ ※ର࿩γεςϜͷݚڀ։ൃΛ͍ͯ͠ΔاۀͰ͢ • ओʹ΍͍ͬͯΔ͜ͱ • αʔόʔαΠυ։ൃ • ͜͜࠷ۙ৮ٕͬͨज़ • Terraform, Ansible, Packer ← Infrasctucture as Codeʂ • Docker, Kubernetes, Apache Mesos ← ϙʔλϏϦςΟ͍͜͞͏ʂ • Python, Scala, javascript(ES6) ← ؔ਺ܕ͏·͔͚͘Δͱؾ͍͍࣋ͪʂ • Finagle, React.js ← ϚΠΫϩαʔϏε͍ͨ͠ʂϦΞΫςΟϒ͍ͨ͠ʂ
3. Copyright © Nextremer, inc. All Rights Reserved. 3 ࠓ೔͓࿩͢͠Δ͜ͱ Nextremer ͷػցֶश؀ڥ Shiva ʹ͍͓ͭͯ࿩͠͠·͢
4. Copyright © Nextremer, inc. All Rights Reserved. 4 ػցֶश؀ڥ Shiva 1. ΦϑΟεʹ͋ΔGPUϚγϯʹ 2. WebͷμογϡϘʔυ͔Β 3. ઐ༻ͷJupyterΛ্ཱͪ͛ΒΕΔ
5. Copyright © Nextremer, inc. All Rights Reserved. 5 Shiva͕ղܾ͢Δ՝୊ ▪ GPUϚγϯΛ࢖͍͍͚ͨͲɾɾɾ • Ϋϥ΢υඅ༻Λؾʹͯ͠ؾܰʹ࢖͑ͳ͍ • ओʹ༧ࢉ෇͖ͷPJҎ֎ɺτϨʔχϯάͱ͔ɺҋݚڀͱ͔ • GPUϚγϯΛෳ਺ϢʔβʔͰڞ༗͢Δͷ΋΍Γʹ͍͘ • ϥΠϒϥϦͷڝ߹ɺ୭͕࢖ͬͯΔ͔Θ͔ΒΜ • GPUϚγϯ্Ͱͷ؀ڥߏங͕େมͩͬͨΓ • શһ͕ৄ͍͠Θ͚͡Όͳ͍
6. Copyright © Nextremer, inc. All Rights Reserved. 6 σϞ ※GIFΞχϝʔγϣϯͰ͢
7. Copyright © Nextremer, inc. All Rights Reserved. 7 Abstract ▪ ϢʔβʔຖͷDockerίϯςφ ▪ શ෦ೖΓ؀ڥ & sudoݖݶ ▪ ϗʔϜdirΛશϚγϯͰڞ༗
8. Copyright © Nextremer, inc. All Rights Reserved. 8 Architecture Overview
9. Copyright © Nextremer, inc. All Rights Reserved. 9 Architecture Overview μογϡϘʔυ͔ΒAPIܦ༝Ͱૢ࡞ ɹɾϦιʔεར༻ঢ়گΛऔಘ ɹɾίϯςφͷىಈɾ࡟আ
10. Copyright © Nextremer, inc. All Rights Reserved. 10 Architecture Overview શ୆ͰϗʔϜσΟϨΫτϦΛڞ༗ ɹɾ෼ࢄετϨʔδΛߏங ɹɾίϯςφىಈ࣌ʹϚ΢ϯτ
11. Copyright © Nextremer, inc. All Rights Reserved. 11 Architecture Overview ▪ ϢʔβʔͱϙʔτͷϚοϐϯά • http://host/user/kazuki.morozumi • Jupyter͕Ϧεϯ͢Δϙʔτ͸ϥϯμϜ ▪ ίϯςφՔಇঢ়گΛެ։͢ΔAPI ▪ tmpnbΛΧελϚΠζ • https://github.com/jupyter/tmpnb
12. Copyright © Nextremer, inc. All Rights Reserved. 12 Infrastructure Overview
13. Copyright © Nextremer, inc. All Rights Reserved. 13 Infrastructure Overview ίϯςφΦʔέετϨʔγϣϯʹ͸ Mesos + MarathonΛར༻
14. Copyright © Nextremer, inc. All Rights Reserved. 14 Infrastructure Overview ▪ Mesos+MarathonΛબ୒ͨ͠ཧ༝ • ಛఆͷΞϓϦΛσʔϞϯͱͯ͠ಈ͔ͤΕ͹े෼ • αʔϏεσΟεΧόϦͱ͔͸͍Βͳ͍ • Kubernetes͸ΦʔόʔεϖοΫ͗͢Δ • ߏ੒ཁૉ͕ଟ͘ෳࡶͰߏஙɺϝϯςφϯε͕େม • ୯७ʹMesos+MarathonΛࢼͯ͠Έ͔ͨͬͨ • ߏங౰࣌͸৘ใ͕͔ͳΓগͳ͔ͬͨ
15. Copyright © Nextremer, inc. All Rights Reserved. 15 Infrastructure Overview CPU Intel Core i7-6700K ϝϞϦ 32GB Disk SSD 500GB GPU GTX-1080 ×2ຕ GPUϚγϯεϖοΫ ݟͨ໨΋େࣄʂ ɹ- ͱΓ͋͑ͣޫΒͤΔ ɹ- ΈΜͳʹݟ͑Δ৔ॴʹஔ͘
16. Copyright © Nextremer, inc. All Rights Reserved. 16 ӡ༻͍ͯͯ͠ىͬͨ͜໰୊ ▪ σΟεΫ্ݶ • ֶशσʔλ਺ඦGBΛΞοϓϩʔυͨ͠ΒσΟεΫ্ݶʹୡͯ͠ఀࢭ • ίϯςφଆ͔ΒsshfsͰࣗ෼ͷPCͷσΟεΫΛϚ΢ϯτͯ͠ճආͨ͠ ▪ ίϯςφ͕๫૸ͨ࣌͠ͷηʔϑςΟʔωοτ • ։ൃͨ͠ίʔυͷόάͰϗετͷCPUɾϝϞϦΛ৯͍௵͢ • ϏΪφʔͷϢʔβʔ͕͜ͷΑ͏ͳόάΛ࡞Γ͕ͪ • ରԠࡦ͸ίϯςφຖʹϦιʔε੍ݶΛ͚ͭΔ • ස౓௿͔ͭӨڹ௿ͳͷͰಛʹݱঢ়͸ରԠͤͣ
17. Copyright © Nextremer, inc. All Rights Reserved. 17 ৄࡉ͸Qiitaͷهࣄ΁ʂ
18. Copyright © Nextremer, inc. All Rights Reserved. 18 NextremerͰ͸͜Μͳ͜ͱΛ΍͍ͬͯ·͢ WBC؍ઓઐ༻ Line bot
19. Copyright © Nextremer, inc. All Rights Reserved. 19 NextremerͰ͸͜Μͳ͜ͱΛ΍͍ͬͯ·͢
20. Copyright © Nextremer, inc. All Rights Reserved. 20 NextremerͰ͸͜Μͳ͜ͱΛ΍͍ͬͯ·͢ ▪ ฐࣾCEO ޲ҪΛίϐʔͨ͠ର࿩γεςϜ • 2017/10 ୈ8ճର࿩γεςϜγϯϙδ΢ϜʹͯσϞൃද
21. Copyright © Nextremer, inc. All Rights Reserved. 21 NextremerͰ͸͜Μͳ͜ͱΛ΍͍ͬͯ·͢