Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ScaleShift-jp-2019-summer
Search
ryo nakamaru
July 17, 2019
Research
1
210
ScaleShift-jp-2019-summer
ScaleShift の機能概要、Kubernetes クラスタとの連携について
ryo nakamaru
July 17, 2019
Tweet
Share
More Decks by ryo nakamaru
See All by ryo nakamaru
AWSで楽をするサービスメッシュ入門/appmesh-trial
pottava
1
1.5k
reinforce-2019-recap-lt
pottava
2
4.1k
Firecracker とは何か/what is Firecracker
pottava
12
5.4k
ハイブリッド並列 on Kubernetes/hybrid-parallel-program-on-kubernetes
pottava
1
430
AWS Fargate + Code 兄弟で始める継続的デリバリー / Continuous Delivery with AWS Fargate and Code brothers
pottava
12
3.2k
Singularity と NVIDIA GPU Cloud で作る ハイブリッド機械学習環境の構築 / Building a hybrid environment for Machine Learning with Singularity and NGC
pottava
3
1.3k
明日から始めるちょい足し λ / get-started-with-aws-lambda
pottava
4
2.5k
NGC と Singularity によるハイブリッド機械学習環境 / A hybrid environment for Machine Learning with NGC and Singularity
pottava
0
490
NGC × Singularity での機械学習環境/MachineLearning environment with NGC and Singularity
pottava
2
780
Other Decks in Research
See All in Research
SSII2025 [SS1] レンズレスカメラ
ssii
PRO
2
980
SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery
satai
3
220
Submeter-level land cover mapping of Japan
satai
3
130
Generative Models 2025
takahashihiroshi
21
12k
なめらかなシステムと運用維持の終わらぬ未来 / dicomo2025_coherently_fittable_system
monochromegane
0
640
研究テーマのデザインと研究遂行の方法論
hisashiishihara
5
1.4k
Towards a More Efficient Reasoning LLM: AIMO2 Solution Summary and Introduction to Fast-Math Models
analokmaus
2
300
データサイエンティストの採用に関するアンケート
datascientistsociety
PRO
0
1k
Computational OT #4 - Gradient flow and diffusion models
gpeyre
0
310
MGDSS:慣性式モーションキャプチャを用いたジェスチャによるドローンの操作 / ec75-yamauchi
yumulab
0
250
Adaptive fusion of multi-modal remote sensing data for optimal sub-field crop yield prediction
satai
3
220
Streamlit 総合解説 ~ PythonistaのためのWebアプリ開発 ~
mickey_kubo
1
950
Featured
See All Featured
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
950
Six Lessons from altMBA
skipperchong
28
3.9k
Optimizing for Happiness
mojombo
379
70k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.4k
Code Reviewing Like a Champion
maltzj
524
40k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
50
5.5k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Build The Right Thing And Hit Your Dates
maggiecrowley
36
2.8k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Practical Orchestrator
shlominoach
189
11k
Transcript
ScaleShift ΦϯϓϨϛε / Ϋϥυ Ͱ࣮ݱ͢Δػցֶशڥ June, 2019
!2 ScaleShift Docker ϕʔεɺΦʔϓϯιʔεͷ Web ΫϥΠΞϯτ ΞϓϦέʔγϣϯͰ͢ • ϞσϧߏஙϑΣʔζ -
NGC / ࣗࣾϦϙδτϦ͔Βػցֶश Docker ΠϝʔδΛϫϯΫϦοΫͰऔಘ - ͦͷҙͷ Docker ΠϝʔδΛ Jupyter notebook ίϯςφͱͯ͠ىಈ • ϞσϧֶशϑΣʔζ - ߏஙʹར༻ͨ͠ϥΠϒϥϦ͝ͱ Docker ΠϝʔδʹݻΊϦϙδτϦอଘ - ΫϦοΫ͚ͩͰ Kubernetes Ϋϥελ / Rescale େنܭࢉλεΫΛૹ৴
جຊతͳಈ͖ !3 How does it work?
!4 ScaleShift ͷىಈ ϩʔΧϧʹ Web αʔόʔ্ཱ͕͕ͪΓ·͢
!5 ػցֶशιϑτΣΞͷΠϯετʔϧ NGC / ϓϥΠϕʔτϨδετϦ ͔ΒϫϯΫϦοΫͰμϯϩʔυ
!6 Jupyter notebook ͰͷϞσϧߏங Jupyter Ͱϥοϓͨ͠ίϯςφ͕͔ΜͨΜʹىಈ ϙʔτ࡞ۀྖҬ ίϯςφ͝ͱʹ ͞Εͨ ΫϦʔϯͳڥ
ɹ.
!7 େنܭࢉͷͨΊͷϥοϐϯά ґଘϥΠϒϥϦιʔείʔυ܈Λ·ͱΊɺͻͱͭͷΠϝʔδʹݻΊ·͢
!8 ࣾΫϥελ / ΫϥυܭࢉλεΫೖ ೖઌʹԠͯ͡ඞཁͳ API ͕࣮ߦ͞Ε·͢ ར༻ϦιʔεྔΛܾΊ ΫϥελλεΫೖ
Kubernetes ࿈ܞ !9 Integration with a kubernetes cluster
!10 ػցֶश ͱ Kubernetes Web ք۾Λத৺ʹίϯςφΦʔέετϨʔγϣϯͷσϑΝΫτʹͳͬͨ k8sɻ ػցֶशͷจ຺Ͱίϯςφར༻͕ΜʹͳΓɺԠ༻ࣄྫ͕૿͍͑ͯ·͢ɻ - NVIDIA
͕ެࣜʹαϙʔτΛද໌ [ GTC 2018 Keynote, March 27 ] - Mercari ML Ops Night Vol.1 [ גࣜձࣾ ϝϧΧϦ / May 23, 2018 ] ɹhttps://mercari.connpass.com/event/85931/presentation/ - Jupyter ͚ͩͰػցֶशΛ࣮αʔϏεల։Ͱ͖Δج൫ [ גࣜձࣾϦΫϧʔτϥΠϑελΠϧ ] ɹhttps://engineer.recruit-lifestyle.co.jp/techblog/2018-10-04-ml-platform/ - KubernetesʹΑΔػցֶशج൫ͷઓ [ גࣜձࣾ Preferred Networks / Dec 4, 2018 ] ɹhttps://www.slideshare.net/pfi/kubernetes-125013757
!11 ScaleShift + Kubernetes ߏྫ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC
DockerHub ϓϥΠϕʔτ ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ೖΓ ϩʔΧϧ
!12 1. ػցֶशιϑτΣΞͷબ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ GUI ͔ΒબͿ͚ͩͰ μϯϩʔυ͕࢝·Γ·͢
!13 2. Ϟσϧߏங ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ͕ ϊʔτϒοΫΛىಈ͠·͢
!14 3. ࣮ߦڥɾೖྗσʔλͷసૹ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ScaleShift ͕෦తʹ ඞཁͳసૹΛߦ͍·͢ ݚڀ / ։ൃνʔϜ
!15 4. େنܭࢉͷ࣮ߦΛࢦࣔ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ Kubernetes ͷ Job ͱͯ͠ ܭࢉ݅Λૹ৴͠·͢
!16 5. େنܭࢉͷ࣮ߦ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub Kubernetes
ϓϥΠϕʔτ ϨδετϦ ݚڀ / ։ൃνʔϜ
!17 6. ܭࢉ݁Ռͷ֬ೝ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ
!18 Kubernetes ઃఆ / λεΫ࣮ߦը໘
ScaleShift ͷઃఆ !19 Configurations
!20 ֎෦࿈ܞ ࿈ܞػೳ ઃఆ NVIDIA GPU CLOUD • NVIDIA ࣾͷཧ͢Δػցֶश
Docker Πϝʔδͷ Ұཡ / ৄࡉใऔಘɺΠϝʔδͷμϯϩʔυ ɹAPI Ωʔ & Ϣʔβઃఆ ϓϥΠϕʔτϨδετϦ • ࣗࣾͰཧ͢Δػցֶश Docker Πϝʔδͷ ҰཡใऔಘɺΠϝʔδͷμϯϩʔυ ɹଓઌ & Ϣʔβઃఆ AWS • ػցֶश Docker Πϝʔδͷμϯϩʔυ • ϩʔΧϧϑΝΠϧγεςϜͱ S3 ؒͷσʔλ࿈ܞ ʢ࣮༧ఆʣ Kubernetes • ࣾΫϥελ / ΫϥυͰͷେنܭࢉ࣮ߦ ɹkubecfg Rescale • Rescale ϓϥοτϑΥʔϜͰͷେنܭࢉ࣮ߦ ɹҬࢦఆ & API Ωʔ
!21 ىಈΦϓγϣϯʢൈਮʣ ઃఆ֓ཁ ॳظ SS_JUPYTER_MINIMUM_PORT ίϯςφͷଓϙʔτಈతׂ։࢝൪߸ ɾ30000 SS_LOG_LEVEL ΞϓϦέʔγϣϯͷϩάग़ྗϨϕϧ ɹwarn
SS_WORKSPACE_HOST_DIR ϗετଆͷ࡞ۀσʔλอଘྖҬ ɹͳ͠ʢࢦఆඞਢʣ SS_NGC_REGISTRY_ENDPOINT NGC ଓઌ ɹhttps://registry.nvidia.com SS_NGC_REGISTRY_USER_NAME NGC Ϣʔβʔ໊ ɹ$oauthtoken SS_RESCALE_SINGULARITY_VERSION Rescale Ͱͷ Singularity ϥϯλΠϜόʔδϣϯ ɹ3.2.0 SS_RESCALE_JOB_WALLTIME Rescale ͰͷλεΫ࣮ߦ࠷େ࣌ؒ ɾ3600 docker-compose.yml ʹઃఆΛهࡌɺىಈͰ͖·͢