Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ScaleShift-jp-2019-summer
Search
ryo nakamaru
July 17, 2019
Research
1
210
ScaleShift-jp-2019-summer
ScaleShift の機能概要、Kubernetes クラスタとの連携について
ryo nakamaru
July 17, 2019
Tweet
Share
More Decks by ryo nakamaru
See All by ryo nakamaru
AWSで楽をするサービスメッシュ入門/appmesh-trial
pottava
1
1.5k
reinforce-2019-recap-lt
pottava
2
4.1k
Firecracker とは何か/what is Firecracker
pottava
12
5.4k
ハイブリッド並列 on Kubernetes/hybrid-parallel-program-on-kubernetes
pottava
1
430
AWS Fargate + Code 兄弟で始める継続的デリバリー / Continuous Delivery with AWS Fargate and Code brothers
pottava
12
3.2k
Singularity と NVIDIA GPU Cloud で作る ハイブリッド機械学習環境の構築 / Building a hybrid environment for Machine Learning with Singularity and NGC
pottava
3
1.3k
明日から始めるちょい足し λ / get-started-with-aws-lambda
pottava
4
2.5k
NGC と Singularity によるハイブリッド機械学習環境 / A hybrid environment for Machine Learning with NGC and Singularity
pottava
0
490
NGC × Singularity での機械学習環境/MachineLearning environment with NGC and Singularity
pottava
2
780
Other Decks in Research
See All in Research
[CV勉強会@関東 CVPR2025] VLM自動運転model S4-Driver
shinkyoto
1
120
近似動的計画入門
mickey_kubo
4
980
EOGS: Gaussian Splatting for Efficient Satellite Image Photogrammetry
satai
4
300
[輪講] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
nk35jk
2
570
Adaptive fusion of multi-modal remote sensing data for optimal sub-field crop yield prediction
satai
3
220
業界横断 副業・兼業者の実態調査
fkske
0
190
クラウドのテレメトリーシステム研究動向2025年
yuukit
3
970
ウッドスタックチャン:木材を用いた小型エージェントロボットの開発と印象評価 / ec75-sato
yumulab
1
420
カスタマーサクセスの視点からAWS Summitの展示を考える~製品開発で活用できる勘所~
masakiokuda
2
110
問いを起点に、社会と共鳴する知を育む場へ
matsumoto_r
PRO
0
410
データサイエンティストの就労意識~2015→2024 一般(個人)会員アンケートより
datascientistsociety
PRO
0
710
Submeter-level land cover mapping of Japan
satai
3
130
Featured
See All Featured
Documentation Writing (for coders)
carmenintech
72
4.9k
Designing for humans not robots
tammielis
253
25k
Git: the NoSQL Database
bkeepers
PRO
430
65k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
The Language of Interfaces
destraynor
158
25k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Rails Girls Zürich Keynote
gr2m
95
14k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Building Applications with DynamoDB
mza
95
6.5k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.5k
Transcript
ScaleShift ΦϯϓϨϛε / Ϋϥυ Ͱ࣮ݱ͢Δػցֶशڥ June, 2019
!2 ScaleShift Docker ϕʔεɺΦʔϓϯιʔεͷ Web ΫϥΠΞϯτ ΞϓϦέʔγϣϯͰ͢ • ϞσϧߏஙϑΣʔζ -
NGC / ࣗࣾϦϙδτϦ͔Βػցֶश Docker ΠϝʔδΛϫϯΫϦοΫͰऔಘ - ͦͷҙͷ Docker ΠϝʔδΛ Jupyter notebook ίϯςφͱͯ͠ىಈ • ϞσϧֶशϑΣʔζ - ߏஙʹར༻ͨ͠ϥΠϒϥϦ͝ͱ Docker ΠϝʔδʹݻΊϦϙδτϦอଘ - ΫϦοΫ͚ͩͰ Kubernetes Ϋϥελ / Rescale େنܭࢉλεΫΛૹ৴
جຊతͳಈ͖ !3 How does it work?
!4 ScaleShift ͷىಈ ϩʔΧϧʹ Web αʔόʔ্ཱ͕͕ͪΓ·͢
!5 ػցֶशιϑτΣΞͷΠϯετʔϧ NGC / ϓϥΠϕʔτϨδετϦ ͔ΒϫϯΫϦοΫͰμϯϩʔυ
!6 Jupyter notebook ͰͷϞσϧߏங Jupyter Ͱϥοϓͨ͠ίϯςφ͕͔ΜͨΜʹىಈ ϙʔτ࡞ۀྖҬ ίϯςφ͝ͱʹ ͞Εͨ ΫϦʔϯͳڥ
ɹ.
!7 େنܭࢉͷͨΊͷϥοϐϯά ґଘϥΠϒϥϦιʔείʔυ܈Λ·ͱΊɺͻͱͭͷΠϝʔδʹݻΊ·͢
!8 ࣾΫϥελ / ΫϥυܭࢉλεΫೖ ೖઌʹԠͯ͡ඞཁͳ API ͕࣮ߦ͞Ε·͢ ར༻ϦιʔεྔΛܾΊ ΫϥελλεΫೖ
Kubernetes ࿈ܞ !9 Integration with a kubernetes cluster
!10 ػցֶश ͱ Kubernetes Web ք۾Λத৺ʹίϯςφΦʔέετϨʔγϣϯͷσϑΝΫτʹͳͬͨ k8sɻ ػցֶशͷจ຺Ͱίϯςφར༻͕ΜʹͳΓɺԠ༻ࣄྫ͕૿͍͑ͯ·͢ɻ - NVIDIA
͕ެࣜʹαϙʔτΛද໌ [ GTC 2018 Keynote, March 27 ] - Mercari ML Ops Night Vol.1 [ גࣜձࣾ ϝϧΧϦ / May 23, 2018 ] ɹhttps://mercari.connpass.com/event/85931/presentation/ - Jupyter ͚ͩͰػցֶशΛ࣮αʔϏεల։Ͱ͖Δج൫ [ גࣜձࣾϦΫϧʔτϥΠϑελΠϧ ] ɹhttps://engineer.recruit-lifestyle.co.jp/techblog/2018-10-04-ml-platform/ - KubernetesʹΑΔػցֶशج൫ͷઓ [ גࣜձࣾ Preferred Networks / Dec 4, 2018 ] ɹhttps://www.slideshare.net/pfi/kubernetes-125013757
!11 ScaleShift + Kubernetes ߏྫ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC
DockerHub ϓϥΠϕʔτ ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ೖΓ ϩʔΧϧ
!12 1. ػցֶशιϑτΣΞͷબ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ GUI ͔ΒબͿ͚ͩͰ μϯϩʔυ͕࢝·Γ·͢
!13 2. Ϟσϧߏங ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ͕ ϊʔτϒοΫΛىಈ͠·͢
!14 3. ࣮ߦڥɾೖྗσʔλͷసૹ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ScaleShift ͕෦తʹ ඞཁͳసૹΛߦ͍·͢ ݚڀ / ։ൃνʔϜ
!15 4. େنܭࢉͷ࣮ߦΛࢦࣔ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ Kubernetes ͷ Job ͱͯ͠ ܭࢉ݅Λૹ৴͠·͢
!16 5. େنܭࢉͷ࣮ߦ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub Kubernetes
ϓϥΠϕʔτ ϨδετϦ ݚڀ / ։ൃνʔϜ
!17 6. ܭࢉ݁Ռͷ֬ೝ ετϨʔδ ཧϊʔυ ܭࢉϊʔυ ࣾωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ
ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ
!18 Kubernetes ઃఆ / λεΫ࣮ߦը໘
ScaleShift ͷઃఆ !19 Configurations
!20 ֎෦࿈ܞ ࿈ܞػೳ ઃఆ NVIDIA GPU CLOUD • NVIDIA ࣾͷཧ͢Δػցֶश
Docker Πϝʔδͷ Ұཡ / ৄࡉใऔಘɺΠϝʔδͷμϯϩʔυ ɹAPI Ωʔ & Ϣʔβઃఆ ϓϥΠϕʔτϨδετϦ • ࣗࣾͰཧ͢Δػցֶश Docker Πϝʔδͷ ҰཡใऔಘɺΠϝʔδͷμϯϩʔυ ɹଓઌ & Ϣʔβઃఆ AWS • ػցֶश Docker Πϝʔδͷμϯϩʔυ • ϩʔΧϧϑΝΠϧγεςϜͱ S3 ؒͷσʔλ࿈ܞ ʢ࣮༧ఆʣ Kubernetes • ࣾΫϥελ / ΫϥυͰͷେنܭࢉ࣮ߦ ɹkubecfg Rescale • Rescale ϓϥοτϑΥʔϜͰͷେنܭࢉ࣮ߦ ɹҬࢦఆ & API Ωʔ
!21 ىಈΦϓγϣϯʢൈਮʣ ઃఆ֓ཁ ॳظ SS_JUPYTER_MINIMUM_PORT ίϯςφͷଓϙʔτಈతׂ։࢝൪߸ ɾ30000 SS_LOG_LEVEL ΞϓϦέʔγϣϯͷϩάग़ྗϨϕϧ ɹwarn
SS_WORKSPACE_HOST_DIR ϗετଆͷ࡞ۀσʔλอଘྖҬ ɹͳ͠ʢࢦఆඞਢʣ SS_NGC_REGISTRY_ENDPOINT NGC ଓઌ ɹhttps://registry.nvidia.com SS_NGC_REGISTRY_USER_NAME NGC Ϣʔβʔ໊ ɹ$oauthtoken SS_RESCALE_SINGULARITY_VERSION Rescale Ͱͷ Singularity ϥϯλΠϜόʔδϣϯ ɹ3.2.0 SS_RESCALE_JOB_WALLTIME Rescale ͰͷλεΫ࣮ߦ࠷େ࣌ؒ ɾ3600 docker-compose.yml ʹઃఆΛهࡌɺىಈͰ͖·͢