Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
ScaleShift-jp-2019-summer
ryo nakamaru
July 17, 2019
Research
1
130
ScaleShift-jp-2019-summer
ScaleShift の機能概要、Kubernetes クラスタとの連携について
ryo nakamaru
July 17, 2019
Tweet
Share
More Decks by ryo nakamaru
See All by ryo nakamaru
pottava
1
680
pottava
2
3.8k
pottava
11
4.1k
pottava
1
270
pottava
12
2.2k
pottava
3
790
pottava
4
1.8k
pottava
0
330
pottava
2
440
Other Decks in Research
See All in Research
ysekky
3
1.4k
aserg_ufmg
2
180
highlow
0
420
racterub
0
150
sei88888
4
300
izunaga
2
2.6k
liberalarts
2
1.4k
noguhiro2002
1
1.2k
mkimura
2
1k
scatterlab
0
1.4k
noteinc
1
51k
ryoherisson
0
790
Featured
See All Featured
stephaniewalter
260
11k
deanohume
295
28k
garrettdimon
287
110k
trishagee
23
2.5k
davidbonilla
70
3.6k
paulrobertlloyd
71
3.6k
cassininazir
347
20k
smashingmag
230
18k
sugarenia
233
850k
trallard
13
710
danielanewman
200
20k
samanthasiow
56
6.3k
Transcript
ScaleShift ΦϯϓϨϛε / Ϋϥυ Ͱ࣮ݱ͢Δػցֶशڥ June, 2019
!2 ScaleShift Docker ϕʔεɺΦʔϓϯιʔεͷ Web ΫϥΠΞϯτ ΞϓϦέʔγϣϯͰ͢ • ϞσϧߏஙϑΣʔζ -
NGC / ࣗࣾϦϙδτϦ͔Βػցֶश Docker ΠϝʔδΛϫϯΫϦοΫͰऔಘ - ͦͷҙͷ Docker ΠϝʔδΛ Jupyter notebook ίϯςφͱͯ͠ىಈ • ϞσϧֶशϑΣʔζ - ߏஙʹར༻ͨ͠ϥΠϒϥϦ͝ͱ Docker ΠϝʔδʹݻΊϦϙδτϦอଘ - ΫϦοΫ͚ͩͰ Kubernetes Ϋϥελ / Rescale େنܭࢉλεΫΛૹ৴
جຊతͳಈ͖ !3 How does it work?
!4 ScaleShift ͷىಈ ϩʔΧϧʹ Web αʔόʔ্ཱ͕͕ͪΓ·͢
!5 ػցֶशιϑτΣΞͷΠϯετʔϧ NGC / ϓϥΠϕʔτϨδετϦ ͔ΒϫϯΫϦοΫͰμϯϩʔυ
!6 Jupyter notebook ͰͷϞσϧߏங Jupyter Ͱϥοϓͨ͠ίϯςφ͕͔ΜͨΜʹىಈ ϙʔτ࡞ۀྖҬ ίϯςφ͝ͱʹ ͞Εͨ ΫϦʔϯͳڥ
ɹ.
!7 େنܭࢉͷͨΊͷϥοϐϯά ґଘϥΠϒϥϦιʔείʔυ܈Λ·ͱΊɺͻͱͭͷΠϝʔδʹݻΊ·͢
!8 ࣾΫϥελ / ΫϥυܭࢉλεΫೖ ೖઌʹԠͯ͡ඞཁͳ API ͕࣮ߦ͞Ε·͢ ར༻ϦιʔεྔΛܾΊ ΫϥελλεΫೖ
Kubernetes ࿈ܞ !9 Integration with a kubernetes cluster
!10 ػցֶश ͱ Kubernetes Web ք۾Λத৺ʹίϯςφΦʔέετϨʔγϣϯͷσϑΝΫτʹͳͬͨ k8sɻ ػցֶशͷจ຺Ͱίϯςφར༻͕ΜʹͳΓɺԠ༻ࣄྫ͕૿͍͑ͯ·͢ɻ - NVIDIA
͕ެࣜʹαϙʔτΛද໌ [ GTC 2018 Keynote, March 27 ] - Mercari ML Ops Night Vol.1 [ גࣜձࣾ ϝϧΧϦ / May 23, 2018 ] ɹhttps://mercari.connpass.com/event/85931/presentation/ - Jupyter ͚ͩͰػցֶशΛ࣮αʔϏεల։Ͱ͖Δج൫ [ גࣜձࣾϦΫϧʔτϥΠϑελΠϧ ] ɹhttps://engineer.recruit-lifestyle.co.jp/techblog/2018-10-04-ml-platform/ - KubernetesʹΑΔػցֶशج൫ͷઓ [ גࣜձࣾ Preferred Networks / Dec 4, 2018 ] ɹhttps://www.slideshare.net/pfi/kubernetes-125013757
!11 ScaleShift + Kubernetes ߏྫ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC
DockerHub ϓϥΠϕʔτ ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ೖΓ ϩʔΧϧ
!12 1. ػցֶशιϑτΣΞͷબ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ GUI ͔ΒબͿ͚ͩͰ μϯϩʔυ͕࢝·Γ·͢
!13 2. Ϟσϧߏங ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ͕ ϊʔτϒοΫΛىಈ͠·͢
!14 3. ࣮ߦڥɾೖྗσʔλͷసૹ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ScaleShift ͕෦తʹ ඞཁͳసૹΛߦ͍·͢ ݚڀ / ։ൃνʔϜ
!15 4. େنܭࢉͷ࣮ߦΛࢦࣔ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ Kubernetes ͷ Job ͱͯ͠ ܭࢉ݅Λૹ৴͠·͢
!16 5. େنܭࢉͷ࣮ߦ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub Kubernetes
ϓϥΠϕʔτ ϨδετϦ ݚڀ / ։ൃνʔϜ
!17 6. ܭࢉ݁Ռͷ֬ೝ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ
!18 Kubernetes ઃఆ / λεΫ࣮ߦը໘
ScaleShift ͷઃఆ !19 Configurations
!20 ֎෦࿈ܞ ࿈ܞػೳ ઃఆ NVIDIA GPU CLOUD • NVIDIA ࣾͷཧ͢Δػցֶश
Docker Πϝʔδͷ Ұཡ / ৄࡉใऔಘɺΠϝʔδͷμϯϩʔυ ɹAPI Ωʔ & Ϣʔβઃఆ ϓϥΠϕʔτϨδετϦ • ࣗࣾͰཧ͢Δػցֶश Docker Πϝʔδͷ ҰཡใऔಘɺΠϝʔδͷμϯϩʔυ ɹଓઌ & Ϣʔβઃఆ AWS • ػցֶश Docker Πϝʔδͷμϯϩʔυ • ϩʔΧϧϑΝΠϧγεςϜͱ S3 ؒͷσʔλ࿈ܞ ʢ࣮༧ఆʣ Kubernetes • ࣾΫϥελ / ΫϥυͰͷେنܭࢉ࣮ߦ ɹkubecfg Rescale • Rescale ϓϥοτϑΥʔϜͰͷେنܭࢉ࣮ߦ ɹҬࢦఆ & API Ωʔ
!21 ىಈΦϓγϣϯʢൈਮʣ ઃఆ֓ཁ ॳظ SS_JUPYTER_MINIMUM_PORT ίϯςφͷଓϙʔτಈతׂ։࢝൪߸ ɾ30000 SS_LOG_LEVEL ΞϓϦέʔγϣϯͷϩάग़ྗϨϕϧ ɹwarn
SS_WORKSPACE_HOST_DIR ϗετଆͷ࡞ۀσʔλอଘྖҬ ɹͳ͠ʢࢦఆඞਢʣ SS_NGC_REGISTRY_ENDPOINT NGC ଓઌ ɹhttps://registry.nvidia.com SS_NGC_REGISTRY_USER_NAME NGC Ϣʔβʔ໊ ɹ$oauthtoken SS_RESCALE_SINGULARITY_VERSION Rescale Ͱͷ Singularity ϥϯλΠϜόʔδϣϯ ɹ3.2.0 SS_RESCALE_JOB_WALLTIME Rescale ͰͷλεΫ࣮ߦ࠷େ࣌ؒ ɾ3600 docker-compose.yml ʹઃఆΛهࡌɺىಈͰ͖·͢