Droot Internals

id:y_uuki
第９回 コンテナ型仮想化の情報交換会@福岡

Transcript

1. %SPPU*OUFSOBMT
   JEZ@VVLJ
   ୈ̕ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ!෱Ԭ
   

   View Slide

2. id:y_uuki
   @y_uuk1
   ͸ͯͳ!ژ౎
   ΢ΣϒΦϖϨʔγϣϯΤϯδχΞ
   

   View Slide

3. IUUQZVVLJIBUFOBCMPHDPNFOUSZESPPU
   

   View Slide

4. TL;DR
   • ιϑτ΢ΣΞґଘ஍ࠈͷղܾͷͨΊʹ
   DockerΛ࢖͍͍ͨ
   • ຊ൪؀ڥͰDockerΛӡ༻͢Δͷ͸ͭΒ͍
   • ʮBuild, Ship, Runʯͱ͍͏ίϯηϓτ͸޷͖
   • DockerΠϝʔδΛS3Λܦ༝ͯ͠഑෍͠ɺ
   chrootͰ࣮ߦ͢Δख๏ͷఏҊ
   

   View Slide

5. Ծ૝Խٕज़
   )8Ծ૝Խ 04Ծ૝Խ
   ,7.
   9FO
   ʜ
   γεςϜίϯςφ
   -9$
   
   ΞϓϦέʔγϣϯ
   ίϯςφ
   %PDLFS
   
   

   View Slide

6. ͳͥ(๻͸)DockerΛ࢖͏ͷ͔
   • ✘ VMΑΓߴ଎
   • ✘ Πϛϡʔλϒϧ
   • ✘ Φʔτεέʔϧ
   • ˚ ϙʔλϏϦςΟ
   • ○ ϓϩάϥϚϒϧͳϗετ؀ڥ
   • ○ ιϑτ΢ΣΞґଘ஍ࠈͷղܾ
   

   View Slide

7. ιϑτ΢ΣΞґଘ஍ࠈ
   • ͋Διϑτ΢ΣΞ͸͍͍ͨͯෳ਺ͷιϑτ
   ΢ΣΞʹґଘ͢Δ
   • ґଘઌͷιϑτ΢ΣΞ΋·ͨෳ਺ͷιϑτ
   ΢ΣΞʹґଘ͢Δ
   • ಉ͡؀ڥΛ࠶ݱ͢Δͷ͕೉͍͠
   • BundlerͳͲΛ࢖ͬͯ΋CͷϥΠϒϥϦʹ
   ґଘ͢Δ͜ͱ΋
   

   View Slide

8. Docker
   • LinuxͷσΟετϦϏϡʔγϣϯ؀ڥ
   ͝ͱݻΊͯΠϝʔδԽ
   • /lib, /usr/bin, /etcͳͲͥΜͿ
   • Linux NamespacesͰಠཱͨ͠؀ڥΛ
   ࡞ͬͯΠϝʔδΛల։
   

   View Slide

9. DockerࠔΔ͜ͱ
   • Docker Engineͷෆ҆ఆ͞
   • ωοτϫʔΫ·ΘΓͷύϑΥʔϚϯεྼԽ
   • ίϯςφͷΰϛ૟আ
   • ίϯςφͷແఀࢭσϓϩΠ
   • ίϯςφͷϩά؅ཧ
   • ίϯςφͷ؂ࢹ
   • ίϯςφͷσόοά
   • Docker Registryͷӡ༻
   

   View Slide

10. chroot
    

    View Slide

11. chroot ☓ Docker ͷΞΠσΞ
    EPDLFSQVMMNZTRM
    $0/5"*/&3@*% EPDLFSDSFBUFNZTRM
    
    EPDLFSFYQPSU$0/5"*/&3@*%PNZTRMUBS
    NZTRMUBSΛ.Z42-Λಈ͔͍ͨ͠ϗετ΁ίϐʔ
    ͢Δɻ
    UBSYG[WBSDPOUBJOFSTNZTRMNZTRMUBS
    TVEPDISPPUWBSDPOUBJOFSTNZTRMNZTRME
    

    View Slide

12. View Slide

13. ࠷ۙͷΞϓϦέʔγϣϯσϓϩΠ
    • git pull͕஗͍
    • σϓϩΠαʔό͔Βͷrsync΋஗͍
    • tarball σϓϩΠ
    • ੒Ռ෺ΛS3ͳͲʹ഑ஔ͠ɺσϓϩΠઌͰ s3 cp
    Ͱμ΢ϯϩʔυͯ͠ల։
    • Serf/Consul
    • σϓϩΠαʔό͔ΒͷSSH͕஗͍ͨΊ
    

    View Slide

14. Droot
    

    View Slide

15. #VJME
    4IJQ
    3VO ESPPUSVO
    EPDLFSCVJME
    ESPPUFYQPSU
    EPDLFS
    EBFNPO
    4UPSBHF
    4
    
    ESPPUEFQMPZ
    BXTTDQ
    BXTTDQ
    

    View Slide

16. %PDLFS %SPPU
    #VJME EPDLFSCVJME EPDLFSCVJME
    3FHJTUSZ
    %PDLFS)VC
    %JTUSJCVUJPO
    ͳΜͰ΋Α͍

    "NB[PO4
    
    'JMF'PSNBU %PDLFSJNBHF
    ͳΜͰ΋Α͍
    FYUBSH[
    
    $POUBJOFS
    -JOVY
    /BNFTQBDFT
    DISPPU
    

    View Slide

17. $ droot export
    • DockerΠϝʔδͷϑΝΠϧγεςϜΛtarܗࣜͰग़ྗ
    • جຊ͸ docker create && docker export
    • gzip / aws cli ͱͷύΠϓʹΑΓɺtar.gzԽͯ͠S3ʹ഑ஔ
    • ϑΝΠϧγεςϜʹdrootઐ༻ͷ؀ڥม਺ϑΝΠϧ
    (/.drootenv) Λ࢓ࠐΉ
    ESPPUFYQPSUEPDLFSpMFTBQQcH[JQDR
    cBXTTDQTCVDLFUBQQUBSH[
    

    View Slide

18. $ droot deploy
    • ඪ४ೖྗ͔ΒtarΞʔΧΠϒΛಡΈࠐΈɺࢦఆ͠
    ͨσΟϨΫτϦʹల։
    • සൟʹߋ৽͞ΕΔίϯςφͷσϓϩΠΛ૝ఆ
    • rsync mode ͱ symlink mode
    BXTTDQTCVDLFUBQQUBSH[c
    HVO[JQDRcESPPUEFQMPZSPPUWBSDPOUBJOFST
    

    View Slide

19. symlink ʹΑΔ atomic deploy
    • σϓϩΠࡁΈͷίϯςφ؀
    ڥΛࠩ͠ସ͑Δඞཁ͕͋Δ
    • https://gist.github.com/
    datagrok/3807742#file-
    symlink-replacement-md
    • symlink Λ rename(2)ɹ
    (mv -T) Ͱ੾Γସ͑Δ͜ͱ
    ʹΑΓΞτϛοΫʹσΟϨ
    ΫτϦΛࠩ͠ସ͑Δ
    
    ᵓᴷᴷBQQBQQENBJO
    ᵓᴷᴷBQQE
    ᵋᴷᴷNBJO
    ᵋᴷᴷCJO
    ᵋᴷᴷCPPU
    ᵋᴷᴷEFW
    
    ᵋᴷᴷCBDLVQ
    ᵋᴷᴷCJO
    ᵋᴷᴷCPPU
    ᵋᴷᴷEFW
    
    

    View Slide

20. $ droot run
    • ࢦఆͨ͠σΟϨΫτϦΛchroot jailͱ࣮ͯ͠ߦ
    • σόΠεϑΝΠϧͷ࡞੒ (/dev/null, /dev/zeroͳͲ)
    • ϗετͷ /etc/group, /etc/resolve.confͳͲΛίϐʔ
    • bind mountͰϗετଆͷ೚ҙͷσΟϨΫτϦΛϚ΢ϯτ
    • Linux capabilities(7) ͰݖݶΛ཈੍
    TVEPESPPUSVODQCJOEWBSMPHSPPU
    ɹɹWBSDPOUBJOFSTBQQDPNNBOE
    

    View Slide

21. chroot(2)
    • ϓϩηεͷϧʔτσΟϨΫτϦΛมߋ
    • ϓϩηεͷઈରύεͷ୳ࡧى఺ͷมߋͷΈ
    • ϓϩηεΛੜ੒ͨ͠Γ͠ͳ͍
    • ΧϨϯτσΟϨΫτϦ͸ͦͷ··ͳͷͰcrhootίʔ
    ϧޙʹchdir(“/“)͢Δ͜ͱ͕ଟ͍
    • jail؀ڥ֎ͷϑΝΠϧ΁షΒΕͨγϯϘϦοΫϦϯ
    Ϋ΁͸ΞΫηεͰ͖ͳ͍
    

    View Slide

22. BindϚ΢ϯτ
    • Linux 2.2͔Βಋೖ
    • σΟϨΫτϦ΍ϑΝΠϧΛଞͷҐஔ΁Ϛ΢ϯτ
    • chroot jail؀ڥ಺͔ΒϚ΢ϯτઌͷϑΝΠϧ΍σΟϨ
    ΫτϦ΁ΞΫηεͰ͖Δ
    • /var/containers/app/var/log ͱ͔ࢀর͢Δͷ໘౗
    • mount -o bind /var/log /var/containers/app/var/log
    • ϗετͷ /var/log Λڞ༗
    

    View Slide

23. LinuxέʔύϏϦςΟ
    • chroot(2)͸ಛݖϓϩηεͰͳ͍ͱίʔϧͰ͖ͳ͍
    • (ݫີʹ͸CAP_SYS_CHROOT)
    • ͔͠͠ɺεʔύʔϢʔβͰಈ͔͢ͷ͸ෆ҆
    • εʔύʔϢʔβͰಈ͔ͭͭ͠ɺcapabilities(7)Ͱඞཁͳ
    ݖݶҎ֎Λམͱ͓ͯ͘͠
    • CAP_CHOWN, CAP_DAC_OVERRIDE,
    CAP_DAC_READ_SEARCH, CAP_FOWNER,
    CAP_SETGID, CAP_SETUID, CAP_NET_BIND_SERVICE
    ΛڐՄ
    

    View Slide

24. Problems
    

    View Slide

25. • Docker (NamespacesΛ࢖ͬͨίϯςφ)΄Ͳͷɹ
    ϙʔλϏϦςΟ͸ͳ͍
    • Dockerίϯςφͷ؀ڥม਺͕Ҿ͖ܧ͕Εͳ͍
    • Dockerίϯςφ্ͷ user/group ͕σϓϩΠઌ
    ϗετʹ͍ͳ͍
    ϙʔλϏϦςΟͷ໰୊
    

    View Slide

26. • ؀ڥม਺͸ϑΝΠϧͱͯ͠ӬଓԽ͞Εͳ͍ͷͰɺ
    Ұ୴ϑΝΠϧʹอଘ
    • droot export ͰҰ୴Dockerίϯςφͱͯ͠ىಈ͞
    ͔ͤͯΒ env ίϚϯυͷ࣮ߦ݁ՌΛ /.drootenv ͱ
    ͯ͠อଘ
    • droot run Ͱ /.drootnenv ΛಡΈͩͯ͠؀ڥม਺Λ
    ෮ݩ
    • droot run —env Ͱ؀ڥม਺ͷ௥Ճɾ্ॻ͖Մೳ
    ؀ڥม਺ͷҾ͖ܧ͗
    

    View Slide

27. • User Namespaces಺Ͱ/etc/groupͳͲΛΈͯඞཁ
    ͳuser/grpupΛࣗಈ࡞੒
    • ϓϩηεπϦʔߏ଄Λ͔͑ͨ͘ͳ͍ͷͰɺclone(2)
    Ͱ͸ͳ͘ chroot(2) ޙʹ unshare(2) ͢Δ
    • clone(2) ͩͱࢠϓϩηεΛੜ੒͢ΔͨΊɺεʔ
    ύʔόΠβϓϩηεͷ഑ԼͰdroot runͨ͠ͱ͖
    ʹɺγάφϧ؅ཧ͕͏·͍͔͘ͳ͍͔΋
    user/groupͷࣗಈ࡞੒(ະ࣮૷)
    

    View Slide

28. • ͜Ε͸͓ͦΒ͘ PID Namespacesͷ࿩
    • https://lwn.net/Articles/532748/
    • Namespaces௚Լͷϓϩηε͕pid 1ͱͯ͠ৼΔ෣͏ඞཁ͕Ͱ
    ͯ͘Δ
    • orphanϓϩηεͷճऩ͢Δඞཁ͕͋Δ
    

    View Slide

29. ίϯςφ͸
    ࣗ෼Ͱ࡞ΕΔ
    

    View Slide

30. (PMBOH
    

    View Slide

31. • github.com/docker/docker/pkg
    • archive, devicemapper, fileutils, mount, symlink…
    • github.com/opencontainers/runc/libcontainer
    • Linux Namespaces·ΘΓ
    • https://github.com/syndtr/gocapability
    • LinuxέʔύϏϦςΟ
    • github.com/docker/engine-api
    • Docker APIΫϥΠΞϯτ
    ίϯςφπʔϧ޲͚ύοέʔδ
    

    View Slide

32. • ࣗ࡞ͷίϯςφπʔϧ Drootͷഎܠͱ࣮૷
    • Build, Ship, RunΛ࣮ݱ͢Δୈ̏ͷιϑτ΢ΣΞ
    • droot export, droot deploy, droot run
    • DockerͰΠϝʔδΛ࡞ͬͯ chroot Ͱ࣮ߦ
    • ϙʔλϏϦςΟͷ໰୊ͱͦͷղܾ
    • ίϯςφ͸ࣗ෼Ͱ࡞ΕΔ
    ·ͱΊ
    

    View Slide

33. github.com/yuuki/droot
    

    View Slide