Rancherでつくる KubernetesのGPUクラスタ

3BODIFS.FFUVQJO,ZPUP 3BODIFSͰͭ͘Δ ,VCFSOFUFTͷ(16Ϋϥελ ࠤʑ໦ਅ໵

8IP Shinya Sasaki Head of Infrastructure Engineering at AlpacaJapan Co.,
Ltd. Osaka, Japan

Ҏલͷൃද

ࠓ೔͸͜ͷลͷ࿩

ઃஔ؀ڥ w ๭σʔληϯλʔ ૔ݿʁ w ϥοΫͱిݯɺΠϯλʔωοτ઀ଓΛ࢖Θͤͯ΋Β͍ͬͯ Δ w ػثͷઃஔɺ-"/݁ઢɺύʔπަ׵౳͸ݱ஍ͷํʹ΍ͬͯ
΋Β͑Δ

ϋʔυ΢ΣΞ w طଘͰ࢖͍ͬͯΔ(16αʔόΛ࠶ηοτΞοϓ͠ɺॱ࣍Ϋ ϥελʹऩ༰ w ࠓޙ૿ઃ༧ఆ

ϚβʔϘʔυ w (16͕ຕૠ͞Δ΋ͷ͕΄͔ʹͳ͔ͬͨΒ͍͠

(16 w (F'PSDF(59 w (F'PSDF359

%JTL w 64#઀ଓͷ44% w ͠ΐͬͪΎ͏յΕΔͷͰ

ిݯ؅ཧ w *1.*ͱ͔J-0ͷΑ͏ͳϦϞʔτ؅ཧػೳ͸ͳ͍ w .FSPTTͰεϚϗͰిݯ0''0/

ඞཁͳλεΫ w 3BODIFS؀ڥͷߏங w (16ϊʔυͷηοτΞοϓखॱͷཱ֬ w ηοτΞοϓͷࣗಈԽ

3BODIFS؀ڥͷߏங

3BODIFS؀ڥͷߏங ׂ Ѫ

(16ϊʔυͷηοτΞοϓखॱͷཱ֬

(16ϊʔυͷηοτΞοϓ w 04Πϯετʔϧ w $6%"Πϯετʔϧ w EPDLFSΠϯετʔϧ w OWJEJBEPDLFSΠϯετʔϧ w
3BODIFSΫϥελʹ+PJO

04Πϯετʔϧ w 6CVOUV

04Πϯετʔϧ w 6CVOUV-54 44%Λೝࣝ͠ͳ͍

04Πϯετʔϧ w 6CVOUV-54 w 6CVOUV-54

04Πϯετʔϧ w 6CVOUV-54 w 6CVOUV-54 Πϯετʔϧ్தͰΤϥʔ

04Πϯετʔϧ w 6CVOUV-54 w 6CVOUV-54 w 6CVOUV

04Πϯετʔϧ w 6CVOUV-54 w 6CVOUV-54 w 6CVOUV Πϯετʔϧ੒ޭ

04Πϯετʔϧ w 6CVOUV-54 w 6CVOUV-54 w 6CVOUV Πϯετʔϧ੒ޭ ͕ɺOWJEJBEPDLFS͕ʹର Ԡ͍ͯ͠ͳ͍ͷͰɺͷ΋ͷ
ΛΠϯετʔϧ͢Δͱ͔ɾɾɾ

04Πϯετʔϧ w 6CVOUV-54 w 6CVOUV-54 w 6CVOUV ͱΓ͋͑ͣɾɾɾ

͜͏͍͏ͷ͕͍΍ͰΫϥ΢υΤϯδχΞʹ ͳͬͨͷͰ͸ͳ͔ͬͨͷ͔ɾɾɾ

͍࢝͟ΊΔͱɾɾɾ w w 6CVOUV-54ϦϦʔε

04Πϯετʔϧ w w 6CVOUV-54ϦϦʔε

͜͏͍͏ͷ͕͍΍Ͱ SZ

OWJEJBTNJ OWJEJBTNJ 5IV4FQ c/7*%*"4.*%SJWFS7FSTJPOc c
c(16/BNF1FSTJTUFODF.c#VT*E%JTQ"c7PMBUJMF6ODPSS&$$c c'BO5FNQ1FSG1XS6TBHF$BQc.FNPSZ6TBHFc(166UJM$PNQVUF.c c c c(F'PSDF(590⒎c0⒎c/"c c$188c.J#.J#c%FGBVMUc c(F'PSDF(590⒎c0⒎c/"c c$188c.J#.J#c%FGBVMUc c(F'PSDF(590⒎c0⒎c/"c c$188c.J#.J#c%FGBVMUc c(F'PSDF(590⒎c"0⒎c/"c c$188c.J#.J#c%FGBVMUc

ηοτΞοϓͷࣗಈԽ

͜Εʂ IUUQLZTNPIBUFOBCMPHKQFOUSZ

؅ཧ༻αʔό

.""4͸Α͔ͬͨ w ͦΜͳʹ೉͘͠ͳ͔ͬͨ w %)$1ͱ͔શ෦؅ཧͰ͖Δ w ϋʔυ΢ΣΞ৘ใ΋ݟΕΔ

͕ɺ໰୊͕ɾɾɾ w ηοτΞοϓதʹQPXFSP⒎͕૸Δ w ϚβʔϘʔυ͕෮ిʹରԠ͍ͯ͠ͳ͍ w ṖͷεΠονͷΈͰ͔͠෮چͰ͖ͳ͍ w ϦϞʔτͰ͸ରԠෆՄ

ṖͷεΠον

݁ہ͖͋ΒΊͯQSFTFFE w %)$1ɺ5'51ɺ1SFTFFEͦΕͧΕΠϯετʔϧ w ςΩετϕʔεͷઃఆϑΝΠϧ w ਓʹҾ͖ܧ͙͜ͱΛߟ͑ΔͱͰ͖Ε͹΍Γͨ͘ͳ͔ͬ ͨɾɾɾ

EIDQEDPOG IPTUOWYM\ IBSEXBSFFUIFSOFUFEFYYYYYY pYFEBEESFTT PQUJPOIPTUOBNFlTFSW ^ IPTUOWYM\ IBSEXBSFFUIFSOFUFEFYYYYZZ pYFEBEESFTT PQUJPOIPTUOBNFTFSW
^ *1ΞυϨεͷ؅ཧͱ͔ͨ͘͠ͳ͔͕ͬͨɺނোͨ͠ ͱ͖ͷަ׵ґཔͱ͔ߟ͑Δͱ."$ΞυϨεʹΑΔݻ ఆ*1؅ཧ͕ඞཁ

QSFTFFEDGH ɿ (OSΠϯετʔϧؔ࿈͸ུ) in-target /bin/mkdir /home/ubuntu/.ssh ;\ in-target /bin/chmod 700
/home/ubuntu/.ssh ;\ in-target /bin/sh -c 'echo "ssh-rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > /home/ ubuntu/.ssh/authorized_keys' ;\ in-target /bin/chown -R ubuntu:ubuntu /home/ubuntu/.ssh ;\ in-target /bin/sh -c 'echo "ubuntu ALL = NOPASSWD: ALL" > /etc/sudoers.d/ubuntu' ;\ in-target /bin/sh -c 'curl -fsSL https://xxxxxxxxxxxxxxxxxxxxxx/install-node.sh | bash' d-i finish-install/reboot_in_progress note

/home/ubuntu/.ssh ;\ in-target /bin/sh -c 'echo "ssh-rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > /home/ ubuntu/.ssh/authorized_keys' ;\ in-target /bin/chown -R ubuntu:ubuntu /home/ubuntu/.ssh ;\ in-target /bin/sh -c 'echo "ubuntu ALL = NOPASSWD: ALL" > /etc/sudoers.d/ubuntu' ;\ in-target /bin/sh -c 'curl -fsSL https://xxxxxxxxxxxxxxxxxxxxxx/install-node.sh | bash' d-i finish-install/reboot_in_progress note 44)ϩάΠϯͰ͖ΔΑ͏ʹ

/home/ubuntu/.ssh ;\ in-target /bin/sh -c 'echo "ssh-rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > /home/ ubuntu/.ssh/authorized_keys' ;\ in-target /bin/chown -R ubuntu:ubuntu /home/ubuntu/.ssh ;\ in-target /bin/sh -c 'echo "ubuntu ALL = NOPASSWD: ALL" > /etc/sudoers.d/ubuntu' ;\ in-target /bin/sh -c 'curl -fsSL https://xxxxxxxxxxxxxxxxxxxxxx/install-node.sh | bash' d-i finish-install/reboot_in_progress note ؅ཧ͠΍͍͢Α͏ʹΠϯε τʔϧεΫϦϓτ͸֎෦ʹ

/home/ubuntu/.ssh ;\ in-target /bin/sh -c 'echo "ssh-rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > /home/ ubuntu/.ssh/authorized_keys' ;\ in-target /bin/chown -R ubuntu:ubuntu /home/ubuntu/.ssh ;\ in-target /bin/sh -c 'echo "ubuntu ALL = NOPASSWD: ALL" > /etc/sudoers.d/ubuntu' ;\ in-target /bin/sh -c 'curl -fsSL https://xxxxxxxxxxxxxxxxxxxxxx/install-node.sh | bash' d-i finish-install/reboot_in_progress note ࠶ىಈ

JOTUBMMOPEFTI apt-get update hostnamectl set-hostname localhost apt-get -y install curl
apt-transport-https ca-certificates gnupg-agent software-properties-common open-iscsi curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" add-apt-repository -y ppa:graphics-drivers curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/ sources.list.d/nvidia-docker.list apt-get update apt-get install -y vim nvidia-docker2 ubuntu-drivers-common nvidia-cuda-toolkit docker-ce docker-ca-cli containerd.io ubuntu-drivers autoinstall if ! grep -e "default-runtime" /etc/docker/daemon.json -e "default-runtime" /etc/docker/daemon.json >/dev/null; then sed -i -e "2i \ \ \ \ \"default-runtime\": \"nvidia\"," /etc/docker/daemon.json ; fi

ηοτΞοϓͷྲྀΕ w ݱ஍ͷํʹ-"/ʹ઀ଓ͠ɺىಈͯ͠΋Β͏ w ࣗಈ04Πϯετʔϧ։࢝ˠ࠶ىಈ w ϦϞʔτ઀ଓ֬ೝ͠ɺखಈͰΫϥελʹ+PJO͢ΔίϚϯυ ࣮ߦ

ΫϥελࢀՃ

ӡ༻։࢝ޙͷ(16ϊʔυͷ໾ׂ w ౰ॳLTͷ؅ཧܥ΋݉Ͷ͍͕ͯͨɺΑ͘ίέΔͷͰ؅ཧܥ ͸&$ʹͯ͠ɺ(16ϊʔυ͸8PSLFSઐ༻ʹͨ͠ w ؅ཧܥίέΔͱ݁ߏ໘౗ɾɾɾ

͜ͷลͷ࿩

؅ཧܥ&$

8PSLFSͱͯ͠ͷ&$

(16ϊʔυ

͜ͷลͷ࿩΋ ·ͨػձ͕͋Ε͹ʂ

͋Γ͕ͱ͏͍͟͝·ͨ͠

Rancherでつくる KubernetesのGPUクラスタ

Rancherでつくる KubernetesのGPUクラスタ

More Decks by sasaki

Other Decks in Technology

Featured

Transcript