Rancherでつくる KubernetesのGPUクラスタ

Transcript

1. 3BODIFS.FFUVQJO,ZPUP
   3BODIFSͰͭ͘Δ
   ,VCFSOFUFTͷ(16Ϋϥελ
   
   ࠤʑ໦ਅ໵
   

   View Slide

2. 8IP
   Shinya Sasaki
   Head of Infrastructure Engineering
   at AlpacaJapan Co., Ltd.
   Osaka, Japan
   

   View Slide

3. Ҏલͷൃද
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. Ҏલͷൃද
   

   View Slide

8. View Slide

9. ࠓ೔͸͜ͷลͷ࿩
   

   View Slide

10. ઃஔ؀ڥ
    w ๭σʔληϯλʔ ૔ݿʁ
    
    w ϥοΫͱిݯɺΠϯλʔωοτ઀ଓΛ࢖Θͤͯ΋Β͍ͬͯ
    Δ
    w ػثͷઃஔɺ-"/݁ઢɺύʔπަ׵౳͸ݱ஍ͷํʹ΍ͬͯ
    ΋Β͑Δ
    

    View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. ϋʔυ΢ΣΞ
    w طଘͰ࢖͍ͬͯΔ(16αʔόΛ࠶ηοτΞοϓ͠ɺॱ࣍Ϋ
    ϥελʹऩ༰
    w ࠓޙ૿ઃ༧ఆ
    

    View Slide

18. ϚβʔϘʔυ
    w (16͕ຕૠ͞Δ΋ͷ͕΄͔ʹͳ͔ͬͨΒ͍͠
    

    View Slide

19. (16
    w (F'PSDF(59
    w (F'PSDF359
    

    View Slide

20. %JTL
    w 64#઀ଓͷ44%
    w ͠ΐͬͪΎ͏յΕΔͷͰ
    

    View Slide

21. ిݯ؅ཧ
    w *1.*ͱ͔J-0ͷΑ͏ͳϦϞʔτ؅ཧػೳ͸ͳ͍
    w .FSPTTͰεϚϗͰిݯ0''0/
    

    View Slide

22. ඞཁͳλεΫ
    w 3BODIFS؀ڥͷߏங
    w (16ϊʔυͷηοτΞοϓखॱͷཱ֬
    w ηοτΞοϓͷࣗಈԽ
    

    View Slide

23. 3BODIFS؀ڥͷߏங
    

    View Slide

24. 3BODIFS؀ڥͷߏங
    ׂ
    Ѫ
    

    View Slide

25. (16ϊʔυͷηοτΞοϓखॱͷཱ֬
    

    View Slide

26. (16ϊʔυͷηοτΞοϓ
    w 04Πϯετʔϧ
    w $6%"Πϯετʔϧ
    w EPDLFSΠϯετʔϧ
    w OWJEJBEPDLFSΠϯετʔϧ
    w 3BODIFSΫϥελʹ+PJO
    

    View Slide

27. 04Πϯετʔϧ
    w 6CVOUV
    

    View Slide

28. 04Πϯετʔϧ
    w 6CVOUV-54 44%Λೝࣝ͠ͳ͍
    

    View Slide

29. 04Πϯετʔϧ
    w 6CVOUV-54
    w 6CVOUV-54
    

    View Slide

30. 04Πϯετʔϧ
    w 6CVOUV-54
    w 6CVOUV-54 Πϯετʔϧ్தͰΤϥʔ
    

    View Slide

31. 04Πϯετʔϧ
    w 6CVOUV-54
    w 6CVOUV-54
    w 6CVOUV
    

    View Slide

32. 04Πϯετʔϧ
    w 6CVOUV-54
    w 6CVOUV-54
    w 6CVOUV Πϯετʔϧ੒ޭ
    

    View Slide

33. 04Πϯετʔϧ
    w 6CVOUV-54
    w 6CVOUV-54
    w 6CVOUV Πϯετʔϧ੒ޭ
    ͕ɺOWJEJBEPDLFS͕ʹର
    Ԡ͍ͯ͠ͳ͍ͷͰɺͷ΋ͷ
    ΛΠϯετʔϧ͢Δͱ͔ɾɾɾ
    

    View Slide

34. 04Πϯετʔϧ
    w 6CVOUV-54
    w 6CVOUV-54
    w 6CVOUV
    ͱΓ͋͑ͣɾɾɾ
    

    View Slide

35. ͜͏͍͏ͷ͕͍΍ͰΫϥ΢υΤϯδχΞʹ
    ͳͬͨͷͰ͸ͳ͔ͬͨͷ͔ɾɾɾ
    

    View Slide

36. ͍࢝͟ΊΔͱɾɾɾ
    w
    w 6CVOUV-54ϦϦʔε
    

    View Slide

37. 04Πϯετʔϧ
    w
    w 6CVOUV-54ϦϦʔε
    

    View Slide

38. ͜͏͍͏ͷ͕͍΍Ͱ SZ
    

    View Slide

39. OWJEJBTNJ
    OWJEJBTNJ
    5IV4FQ
    
    c/7*%*"4.*%SJWFS7FSTJPOc
    c
    c(16/BNF1FSTJTUFODF.c#VT*E%JTQ"c7PMBUJMF6ODPSS&$$c
    c'BO5FNQ1FSG1XS6TBHF$BQc.FNPSZ6TBHFc(166UJM$PNQVUF.c
    cc
    c(F'PSDF(590⒎c0⒎c/"c
    c$188c.J#.J#c%FGBVMUc
    
    c(F'PSDF(590⒎c0⒎c/"c
    c$188c.J#.J#c%FGBVMUc
    
    c(F'PSDF(590⒎c0⒎c/"c
    c$188c.J#.J#c%FGBVMUc
    
    c(F'PSDF(590⒎c"0⒎c/"c
    c$188c.J#.J#c%FGBVMUc
    
    

    View Slide

40. ηοτΞοϓͷࣗಈԽ
    

    View Slide

41. ͜Εʂ
    IUUQLZTNPIBUFOBCMPHKQFOUSZ
    

    View Slide

42. ؅ཧ༻αʔό
    

    View Slide

43. View Slide

44. .""4͸Α͔ͬͨ
    w ͦΜͳʹ೉͘͠ͳ͔ͬͨ
    w %)$1ͱ͔શ෦؅ཧͰ͖Δ
    w ϋʔυ΢ΣΞ৘ใ΋ݟΕΔ
    

    View Slide

45. ͕ɺ໰୊͕ɾɾɾ
    w ηοτΞοϓதʹQPXFSP⒎͕૸Δ
    w ϚβʔϘʔυ͕෮ిʹରԠ͍ͯ͠ͳ͍
    w ṖͷεΠονͷΈͰ͔͠෮چͰ͖ͳ͍
    w ϦϞʔτͰ͸ରԠෆՄ
    

    View Slide

46. ṖͷεΠον
    

    View Slide

47. View Slide

48. ݁ہ͖͋ΒΊͯQSFTFFE
    w %)$1ɺ5'51ɺ1SFTFFEͦΕͧΕΠϯετʔϧ
    w ςΩετϕʔεͷઃఆϑΝΠϧ
    w ਓʹҾ͖ܧ͙͜ͱΛߟ͑ΔͱͰ͖Ε͹΍Γͨ͘ͳ͔ͬ
    ͨɾɾɾ
    

    View Slide

49. EIDQEDPOG
    IPTUOWYM\
    IBSEXBSFFUIFSOFUFEFYYYYYY
    pYFEBEESFTT
    PQUJPOIPTUOBNFlTFSW
    ^
    IPTUOWYM\
    IBSEXBSFFUIFSOFUFEFYYYYZZ
    pYFEBEESFTT
    PQUJPOIPTUOBNFTFSW
    ^
    
    
    *1ΞυϨεͷ؅ཧͱ͔ͨ͘͠ͳ͔͕ͬͨɺނোͨ͠
    ͱ͖ͷަ׵ґཔͱ͔ߟ͑Δͱ."$ΞυϨεʹΑΔݻ
    ఆ*1؅ཧ͕ඞཁ
    

    View Slide

50. View Slide

51. QSFTFFEDGH
    ɿ (OSΠϯετʔϧؔ࿈͸ུ)
    in-target /bin/mkdir /home/ubuntu/.ssh ;\
    in-target /bin/chmod 700 /home/ubuntu/.ssh ;\
    in-target /bin/sh -c 'echo "ssh-rsa
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > /home/
    ubuntu/.ssh/authorized_keys' ;\
    in-target /bin/chown -R ubuntu:ubuntu /home/ubuntu/.ssh ;\
    in-target /bin/sh -c 'echo "ubuntu ALL = NOPASSWD: ALL" > /etc/sudoers.d/ubuntu' ;\
    in-target /bin/sh -c 'curl -fsSL https://xxxxxxxxxxxxxxxxxxxxxx/install-node.sh | bash'
    d-i finish-install/reboot_in_progress note
    

    View Slide

52. QSFTFFEDGH
    ɿ (OSΠϯετʔϧؔ࿈͸ུ)
    in-target /bin/mkdir /home/ubuntu/.ssh ;\
    in-target /bin/chmod 700 /home/ubuntu/.ssh ;\
    in-target /bin/sh -c 'echo "ssh-rsa
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > /home/
    ubuntu/.ssh/authorized_keys' ;\
    in-target /bin/chown -R ubuntu:ubuntu /home/ubuntu/.ssh ;\
    in-target /bin/sh -c 'echo "ubuntu ALL = NOPASSWD: ALL" > /etc/sudoers.d/ubuntu' ;\
    in-target /bin/sh -c 'curl -fsSL https://xxxxxxxxxxxxxxxxxxxxxx/install-node.sh | bash'
    d-i finish-install/reboot_in_progress note
    44)ϩάΠϯͰ͖ΔΑ͏ʹ
    

    View Slide

53. QSFTFFEDGH
    ɿ (OSΠϯετʔϧؔ࿈͸ུ)
    in-target /bin/mkdir /home/ubuntu/.ssh ;\
    in-target /bin/chmod 700 /home/ubuntu/.ssh ;\
    in-target /bin/sh -c 'echo "ssh-rsa
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > /home/
    ubuntu/.ssh/authorized_keys' ;\
    in-target /bin/chown -R ubuntu:ubuntu /home/ubuntu/.ssh ;\
    in-target /bin/sh -c 'echo "ubuntu ALL = NOPASSWD: ALL" > /etc/sudoers.d/ubuntu' ;\
    in-target /bin/sh -c 'curl -fsSL https://xxxxxxxxxxxxxxxxxxxxxx/install-node.sh | bash'
    d-i finish-install/reboot_in_progress note
    ؅ཧ͠΍͍͢Α͏ʹΠϯε
    τʔϧεΫϦϓτ͸֎෦ʹ
    

    View Slide

54. QSFTFFEDGH
    ɿ (OSΠϯετʔϧؔ࿈͸ུ)
    in-target /bin/mkdir /home/ubuntu/.ssh ;\
    in-target /bin/chmod 700 /home/ubuntu/.ssh ;\
    in-target /bin/sh -c 'echo "ssh-rsa
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > /home/
    ubuntu/.ssh/authorized_keys' ;\
    in-target /bin/chown -R ubuntu:ubuntu /home/ubuntu/.ssh ;\
    in-target /bin/sh -c 'echo "ubuntu ALL = NOPASSWD: ALL" > /etc/sudoers.d/ubuntu' ;\
    in-target /bin/sh -c 'curl -fsSL https://xxxxxxxxxxxxxxxxxxxxxx/install-node.sh | bash'
    d-i finish-install/reboot_in_progress note ࠶ىಈ
    

    View Slide

55. JOTUBMMOPEFTI
    apt-get update
    hostnamectl set-hostname localhost
    apt-get -y install curl apt-transport-https ca-certificates gnupg-agent software-properties-common open-iscsi
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
    add-apt-repository -y ppa:graphics-drivers
    curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
    curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/
    sources.list.d/nvidia-docker.list
    apt-get update
    apt-get install -y vim nvidia-docker2 ubuntu-drivers-common nvidia-cuda-toolkit docker-ce docker-ca-cli
    containerd.io
    ubuntu-drivers autoinstall
    if ! grep -e "default-runtime" /etc/docker/daemon.json -e "default-runtime" /etc/docker/daemon.json >/dev/null;
    then sed -i -e "2i \ \ \ \ \"default-runtime\": \"nvidia\"," /etc/docker/daemon.json ; fi
    

    View Slide

56. ηοτΞοϓͷྲྀΕ
    w ݱ஍ͷํʹ-"/ʹ઀ଓ͠ɺىಈͯ͠΋Β͏
    w ࣗಈ04Πϯετʔϧ։࢝ˠ࠶ىಈ
    w ϦϞʔτ઀ଓ֬ೝ͠ɺखಈͰΫϥελʹ+PJO͢ΔίϚϯυ
    ࣮ߦ
    

    View Slide

57. ΫϥελࢀՃ
    

    View Slide

58. ӡ༻։࢝ޙͷ(16ϊʔυͷ໾ׂ
    w ౰ॳLTͷ؅ཧܥ΋݉Ͷ͍͕ͯͨɺΑ͘ίέΔͷͰ؅ཧܥ
    ͸&$ʹͯ͠ɺ(16ϊʔυ͸8PSLFSઐ༻ʹͨ͠
    w ؅ཧܥίέΔͱ݁ߏ໘౗ɾɾɾ
    

    View Slide

59. ͜ͷลͷ࿩
    

    View Slide

60. ؅ཧܥ&$
    

    View Slide

61. 8PSLFSͱͯ͠ͷ&$
    

    View Slide

62. (16ϊʔυ
    

    View Slide

63. View Slide

64. ͜ͷลͷ࿩΋
    ·ͨػձ͕͋Ε͹ʂ
    

    View Slide

65. ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide