Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ScaleShift-jp-2019-summer

1e5a15f4dc65c207a04a1e82a3f92e92?s=47 ryo nakamaru
July 17, 2019
Research
1
150

 ScaleShift-jp-2019-summer

ScaleShift の機能概要、Kubernetes クラスタとの連携について

1e5a15f4dc65c207a04a1e82a3f92e92?s=128

ryo nakamaru

July 17, 2019
Tweet

More Decks by ryo nakamaru

See All by ryo nakamaru
AWSで楽をするサービスメッシュ入門/appmesh-trial
1e5a15f4dc65c207a04a1e82a3f92e92?s=48 pottava
1
850
reinforce-2019-recap-lt
1e5a15f4dc65c207a04a1e82a3f92e92?s=48 pottava
2
4k
Firecracker とは何か/what is Firecracker
1e5a15f4dc65c207a04a1e82a3f92e92?s=48 pottava
11
4.5k
ハイブリッド並列 on Kubernetes/hybrid-parallel-program-on-kubernetes
1e5a15f4dc65c207a04a1e82a3f92e92?s=48 pottava
1
290
AWS Fargate + Code 兄弟で始める継続的デリバリー / Continuous Delivery with AWS Fargate and Code brothers
1e5a15f4dc65c207a04a1e82a3f92e92?s=48 pottava
12
2.6k
Singularity と NVIDIA GPU Cloud で作る ハイブリッド機械学習環境の構築 / Building a hybrid environment for Machine Learning with Singularity and NGC
1e5a15f4dc65c207a04a1e82a3f92e92?s=48 pottava
3
930
明日から始めるちょい足し λ / get-started-with-aws-lambda
1e5a15f4dc65c207a04a1e82a3f92e92?s=48 pottava
4
2k
NGC と Singularity によるハイブリッド機械学習環境 / A hybrid environment for Machine Learning with NGC and Singularity
1e5a15f4dc65c207a04a1e82a3f92e92?s=48 pottava
0
380
NGC × Singularity での機械学習環境/MachineLearning environment with NGC and Singularity
1e5a15f4dc65c207a04a1e82a3f92e92?s=48 pottava
2
500

Other Decks in Research

See All in Research
Recent Findings on Density-Ratio Approaches in Machine Learning
Bb6c3fc8c577710c72d03aeb4fa56bf6?s=48 masakat0
0
220
論文読み会資料 When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism
E054bbb78a219fbddbac0bc492aed606?s=48 sensetime_japan
1
750
第8回チャンピオンズミーティング・サジタリウス杯ラウンド2集計 / Umamusume Sagittarius 2021 Round2
9782dd351a8f1737d4ef02d839e821f9?s=48 kitachan_black
0
1.7k
深層学習によるセマンティックセグメンテーションとその最新動向
C97d86982c70a66b42b4c5b4b004874f?s=48 hf149
0
910
まだ生態学に本格導入されていない統計的因果推論手法の紹介：傾向スコア、回帰分断デザイン、操作変数法を中心に
1e65da5f023dc428b8becced82bad823?s=48 takehikoihayashi
2
850
Collaborative editing through a databases lens
0d4ef9af8e4f0cf5c162b48ba24faea6?s=48 ept
0
240
OSSベースでのRパッケージ開発のすすめ / rjpusers2021rpkgdev
D12a80cab206033a820ccff8319f957b?s=48 s_uryu
0
580
log4shell注意喚起
02335420aa18081affbcb8974de06429?s=48 connect24h
1
2.2k
Adversarial Training
310b7634b5b2a5926a6987720c884170?s=48 hirokiadachi
3
1.3k
第9回チャンピオンズミーティング・カプリコーン杯ラウンド2集計 / Umamusume Capricorn 2022 Round2
9782dd351a8f1737d4ef02d839e821f9?s=48 kitachan_black
0
1.6k
機械学習ベースの動画像処理における近似計算手法の検討 (CPSY 2022/03)
290a66d9796825297c2cdb438a372f6a?s=48 hashi0203
0
170
データサイエンティストと博士の専門性
E0e299d5e09e4726e1a6c808eb58b7a3?s=48 mtakano
1
130

Featured

See All Featured
Easily Structure & Communicate Ideas using Wireframe
0c6fd6a08d0f898159cda7cafcacf07f?s=48 afnizarnur
181
15k
Thoughts on Productivity
A16eb159db895e3b01d3dc95767ad595?s=48 jonyablonski
43
2.2k
jQuery: Nuts, Bolts and Bling
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
56
6.4k
Stop Working from a Prison Cell
C1daf20e5c49ff910745198ef9869ac2?s=48 hatefulcrawdad
261
17k
Facilitating Awesome Meetings
245cee81a9c424266e5e401d844ea881?s=48 lara
29
3.9k
Java REST API Framework Comparison - PWX 2021
72a2082c6a4dd79ad68befb3db911616?s=48 mraible
PRO
11
4.6k
Practical Orchestrator
168ccec72eee0530b818d44f3fedaacf?s=48 shlominoach
178
8.6k
Statistics for Hackers
56c4053438af8e8b90d6f53cbb7573be?s=48 jakevdp
781
210k
The MySQL Ecosystem @ GitHub 2015
Be9caeb9d4ef9944d151af909063ed6e?s=48 samlambert
238
11k
Streamline your AJAX requests with AmplifyJS and jQuery
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
125
8.5k
How to name files
0a4f62e90c976eeb44d33add75cca5af?s=48 jennybc
39
58k
Design by the Numbers
027d1ebf66cc039a0bd3b55eeadbe75d?s=48 sachag
271
17k

Transcript

  1. ScaleShift ΦϯϓϨϛε / Ϋϥ΢υ Ͱ࣮ݱ͢Δػցֶश؀ڥ June, 2019

  2. !2 ScaleShift Docker ϕʔεɺΦʔϓϯιʔεͷ Web ΫϥΠΞϯτ ΞϓϦέʔγϣϯͰ͢ • ϞσϧߏஙϑΣʔζ -

    NGC / ࣗࣾϦϙδτϦ͔Βػցֶश Docker ΠϝʔδΛϫϯΫϦοΫͰऔಘ - ͦͷ೚ҙͷ Docker ΠϝʔδΛ Jupyter notebook ίϯςφͱͯ͠ىಈ • ϞσϧֶशϑΣʔζ - ߏஙʹར༻ͨ͠ϥΠϒϥϦ͝ͱ Docker ΠϝʔδʹݻΊϦϙδτϦ΁อଘ - ΫϦοΫ͚ͩͰ Kubernetes Ϋϥελ / Rescale ΁େن໛ܭࢉλεΫΛૹ৴

  3. جຊతͳಈ͖ !3 How does it work?

  4. !4 ScaleShift ͷىಈ ϩʔΧϧʹ Web αʔόʔ্ཱ͕͕ͪΓ·͢

  5. !5 ػցֶशιϑτ΢ΣΞͷΠϯετʔϧ NGC / ϓϥΠϕʔτϨδετϦ ͔ΒϫϯΫϦοΫͰμ΢ϯϩʔυ

  6. !6 Jupyter notebook ͰͷϞσϧߏங Jupyter Ͱϥοϓͨ͠ίϯςφ͕͔ΜͨΜʹىಈ ϙʔτ΋࡞ۀྖҬ΋ ίϯςφ͝ͱʹ ෼཭͞Εͨ ΫϦʔϯͳ؀ڥ

    ɹ.

  7. !7 େن໛ܭࢉͷͨΊͷϥοϐϯά ґଘϥΠϒϥϦ΍ιʔείʔυ܈Λ·ͱΊɺͻͱͭͷΠϝʔδʹݻΊ·͢

  8. !8 ࣾ಺Ϋϥελ / Ϋϥ΢υ΁ܭࢉλεΫ౤ೖ ౤ೖઌʹԠͯ͡ඞཁͳ API ͕࣮ߦ͞Ε·͢ ར༻ϦιʔεྔΛܾΊ Ϋϥελ΁λεΫ౤ೖ

  9. Kubernetes ࿈ܞ !9 Integration with a kubernetes cluster

  10. !10 ػցֶश ͱ Kubernetes Web ք۾Λத৺ʹίϯςφΦʔέετϨʔγϣϯͷσϑΝΫτʹͳͬͨ k8sɻ ػցֶशͷจ຺Ͱ΋ίϯςφར༻͕੝ΜʹͳΓɺԠ༻ࣄྫ͕૿͍͑ͯ·͢ɻ - NVIDIA

    ͕ެࣜʹαϙʔτΛද໌ [ GTC 2018 Keynote, March 27 ] - Mercari ML Ops Night Vol.1 [ גࣜձࣾ ϝϧΧϦ / May 23, 2018 ] ɹhttps://mercari.connpass.com/event/85931/presentation/ - Jupyter ͚ͩͰػցֶशΛ࣮αʔϏεల։Ͱ͖Δج൫ [ גࣜձࣾϦΫϧʔτϥΠϑελΠϧ ] ɹhttps://engineer.recruit-lifestyle.co.jp/techblog/2018-10-04-ml-platform/ - KubernetesʹΑΔػցֶशج൫΁ͷ௅ઓ [ גࣜձࣾ Preferred Networks / Dec 4, 2018 ] ɹhttps://www.slideshare.net/pfi/kubernetes-125013757

  11. !11 ScaleShift + Kubernetes ߏ੒ྫ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC

    DockerHub ϓϥΠϕʔτ ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ೖΓ ϩʔΧϧ୺຤

  12. !12 1. ػցֶशιϑτ΢ΣΞͷબ୒ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ GUI ͔ΒબͿ͚ͩͰ μ΢ϯϩʔυ͕࢝·Γ·͢

  13. !13 2. Ϟσϧߏங ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ͕ ϊʔτϒοΫΛىಈ͠·͢

  14. !14 3. ࣮ߦ؀ڥɾೖྗσʔλͷసૹ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ScaleShift ͕಺෦తʹ ඞཁͳసૹΛߦ͍·͢ ݚڀ / ։ൃνʔϜ

  15. !15 4. େن໛ܭࢉͷ࣮ߦΛࢦࣔ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ Kubernetes ͷ Job ͱͯ͠ ܭࢉ৚݅Λૹ৴͠·͢

  16. !16 5. େن໛ܭࢉͷ࣮ߦ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub Kubernetes

    ϓϥΠϕʔτ ϨδετϦ ݚڀ / ։ൃνʔϜ

  17. !17 6. ܭࢉ݁Ռͷ֬ೝ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ

  18. !18 Kubernetes ઃఆ / λεΫ࣮ߦը໘

  19. ScaleShift ͷઃఆ !19 Configurations

  20. !20 ֎෦࿈ܞ ࿈ܞػೳ ઃఆ஋ NVIDIA GPU CLOUD • NVIDIA ࣾͷ؅ཧ͢Δػցֶश

    Docker Πϝʔδͷ Ұཡ / ৄࡉ৘ใऔಘɺΠϝʔδͷμ΢ϯϩʔυ ɹAPI Ωʔ & Ϣʔβઃఆ ϓϥΠϕʔτϨδετϦ • ࣗࣾͰ؅ཧ͢Δػցֶश Docker Πϝʔδͷ Ұཡ৘ใऔಘɺΠϝʔδͷμ΢ϯϩʔυ ɹ઀ଓઌ & Ϣʔβઃఆ AWS • ػցֶश Docker Πϝʔδͷμ΢ϯϩʔυ • ϩʔΧϧϑΝΠϧγεςϜͱ S3 ؒͷσʔλ࿈ܞ ʢ࣮૷༧ఆʣ Kubernetes • ࣾ಺Ϋϥελ / Ϋϥ΢υͰͷେن໛ܭࢉ࣮ߦ ɹkubecfg Rescale • Rescale ϓϥοτϑΥʔϜͰͷେن໛ܭࢉ࣮ߦ ɹ஍Ҭࢦఆ & API Ωʔ

  21. !21 ىಈΦϓγϣϯʢൈਮʣ ઃఆ֓ཁ ॳظ஋ SS_JUPYTER_MINIMUM_PORT ίϯςφ΁ͷ઀ଓϙʔτಈతׂ౰։࢝൪߸ ɾ30000 SS_LOG_LEVEL ΞϓϦέʔγϣϯͷϩάग़ྗϨϕϧ ɹwarn

    SS_WORKSPACE_HOST_DIR ϗετଆͷ࡞ۀσʔλอଘྖҬ ɹͳ͠ʢࢦఆඞਢʣ SS_NGC_REGISTRY_ENDPOINT NGC ઀ଓઌ ɹhttps://registry.nvidia.com SS_NGC_REGISTRY_USER_NAME NGC Ϣʔβʔ໊ ɹ$oauthtoken SS_RESCALE_SINGULARITY_VERSION Rescale Ͱͷ Singularity ϥϯλΠϜόʔδϣϯ ɹ3.2.0 SS_RESCALE_JOB_WALLTIME Rescale ͰͷλεΫ࣮ߦ࠷େ࣌ؒ ɾ3600 docker-compose.yml ʹઃఆΛهࡌɺىಈͰ͖·͢