Droot Internals

id:y_uuki
第９回 コンテナ型仮想化の情報交換会@福岡

Transcript

1. %SPPU
   *OUFSOBMT JEZ@VVLJ
   ୈ̕ճ
   ίϯςφܕԾ૝Խͷ৘ใަ׵ձ!෱Ԭ
2. id:y_uuki @y_uuk1 ͸ͯͳ!ژ౎
   ΢ΣϒΦϖϨʔγϣϯΤϯδχΞ
3. IUUQZVVLJIBUFOBCMPHDPNFOUSZESPPU
4. TL;DR • ιϑτ΢ΣΞґଘ஍ࠈͷղܾͷͨΊʹ DockerΛ࢖͍͍ͨ • ຊ൪؀ڥͰDockerΛӡ༻͢Δͷ͸ͭΒ͍ • ʮBuild, Ship, Runʯͱ͍͏ίϯηϓτ͸޷͖ • DockerΠϝʔδΛS3Λܦ༝ͯ͠഑෍͠ɺ chrootͰ࣮ߦ͢Δख๏ͷఏҊ
5. Ծ૝Խٕज़ )8Ծ૝Խ 04Ծ૝Խ ,7.
   9FO
   ʜ γεςϜίϯςφ
   -9$ ΞϓϦέʔγϣϯ ίϯςφ
   %PDLFS
6. ͳͥ(๻͸)DockerΛ࢖͏ͷ͔ • ✘ VMΑΓߴ଎ • ✘ Πϛϡʔλϒϧ • ✘ Φʔτεέʔϧ • ˚ ϙʔλϏϦςΟ • ◦ ϓϩάϥϚϒϧͳϗετ؀ڥ • ◦ ιϑτ΢ΣΞґଘ஍ࠈͷղܾ
7. ιϑτ΢ΣΞґଘ஍ࠈ • ͋Διϑτ΢ΣΞ͸͍͍ͨͯෳ਺ͷιϑτ ΢ΣΞʹґଘ͢Δ • ґଘઌͷιϑτ΢ΣΞ΋·ͨෳ਺ͷιϑτ ΢ΣΞʹґଘ͢Δ • ಉ͡؀ڥΛ࠶ݱ͢Δͷ͕೉͍͠ • BundlerͳͲΛ࢖ͬͯ΋CͷϥΠϒϥϦʹ ґଘ͢Δ͜ͱ΋
8. Docker • LinuxͷσΟετϦϏϡʔγϣϯ؀ڥ ͝ͱݻΊͯΠϝʔδԽ • /lib, /usr/bin, /etcͳͲͥΜͿ • Linux NamespacesͰಠཱͨ͠؀ڥΛ ࡞ͬͯΠϝʔδΛల։
9. DockerࠔΔ͜ͱ • Docker Engineͷෆ҆ఆ͞ • ωοτϫʔΫ·ΘΓͷύϑΥʔϚϯεྼԽ • ίϯςφͷΰϛ૟আ • ίϯςφͷແఀࢭσϓϩΠ • ίϯςφͷϩά؅ཧ • ίϯςφͷ؂ࢹ • ίϯςφͷσόοά • Docker Registryͷӡ༻
10. chroot
11. chroot ☓ Docker ͷΞΠσΞ 
    EPDLFS
    QVMM
    NZTRM
    
    $0/5"*/&3@*% EPDLFS
    DSFBUF
    NZTRM
    
    EPDLFS
    FYQPSU
    $0/5"*/&3@*%
    P
    NZTRMUBS
    
    NZTRMUBS
    Λ.Z42-Λಈ͔͍ͨ͠ϗετ΁ίϐʔ ͢Δɻ
    
    UBS
    YG[
    WBSDPOUBJOFSTNZTRMNZTRMUBS
    
    
    TVEP
    DISPPU
    WBSDPOUBJOFSTNZTRM
    NZTRME
13. ࠷ۙͷΞϓϦέʔγϣϯσϓϩΠ • git pull͕஗͍ • σϓϩΠαʔό͔Βͷrsync΋஗͍ • tarball σϓϩΠ • ੒Ռ෺ΛS3ͳͲʹ഑ஔ͠ɺσϓϩΠઌͰ s3 cp Ͱμ΢ϯϩʔυͯ͠ల։ • Serf/Consul • σϓϩΠαʔό͔ΒͷSSH͕஗͍ͨΊ
14. Droot
15. #VJME 4IJQ 3VO ESPPU
    SVO
    EPDLFS
    CVJME ESPPU
    FYQPSU EPDLFS
    EBFNPO 4UPSBHF
    
    4 ESPPU
    
    EFQMPZ BXT
    T
    DQ BXT
    T
    DQ
16. %PDLFS %SPPU #VJME EPDLFS
    CVJME EPDLFS
    CVJME 3FHJTUSZ %PDLFS)VC %JTUSJCVUJPO ͳΜͰ΋Α͍
 "NB[PO
    4 'JMF
    'PSNBU %PDLFS
    JNBHF ͳΜͰ΋Α͍
    FY
    UBSH[ $POUBJOFS -JOVY
    /BNFTQBDFT DISPPU
17. $ droot export • DockerΠϝʔδͷϑΝΠϧγεςϜΛtarܗࣜͰग़ྗ • جຊ͸ docker create && docker export • gzip / aws cli ͱͷύΠϓʹΑΓɺtar.gzԽͯ͠S3ʹ഑ஔ • ϑΝΠϧγεςϜʹdrootઐ༻ͷ؀ڥม਺ϑΝΠϧ (/.drootenv) Λ࢓ࠐΉ 
    ESPPU
    FYQPSU
    EPDLFSpMFTBQQ
    c
    H[JQ
    DR
    
    c
    BXT
    T
    DQ
    
    TCVDLFUBQQUBSH[
18. $ droot deploy • ඪ४ೖྗ͔ΒtarΞʔΧΠϒΛಡΈࠐΈɺࢦఆ͠ ͨσΟϨΫτϦʹల։ • සൟʹߋ৽͞ΕΔίϯςφͷσϓϩΠΛ૝ఆ • rsync mode ͱ symlink mode 
    BXT
    T
    DQ
    TCVDLFUBQQUBSH[
    
    c
    HVO[JQ
    DR
    c
    ESPPU
    EFQMPZ
    SPPU
    WBSDPOUBJOFST
19. symlink ʹΑΔ atomic deploy • σϓϩΠࡁΈͷίϯςφ؀ ڥΛࠩ͠ସ͑Δඞཁ͕͋Δ • https://gist.github.com/ datagrok/3807742#file- symlink-replacement-md • symlink Λ rename(2)ɹ (mv -T) Ͱ੾Γସ͑Δ͜ͱ ʹΑΓΞτϛοΫʹσΟϨ ΫτϦΛࠩ͠ସ͑Δ 
    ᵓᴷᴷ
    BQQ
    
    BQQENBJO
    ᵓᴷᴷ
    BQQE
    
    
    
    
    ᵋᴷᴷ
    NBJO
    
    
    
    
    
    
    
    
    ᵋᴷᴷ
    CJO
    
    
    
    
    
    
    
    
    ᵋᴷᴷ
    CPPU
    
    
    
    
    
    
    
    
    ᵋᴷᴷ
    EFW
    
    
    
    
    
    
    
    
    
    
    
    
    ᵋᴷᴷ
    CBDLVQ
    
    
    
    
    
    
    
    
    ᵋᴷᴷ
    CJO
    
    
    
    
    
    
    
    
    ᵋᴷᴷ
    CPPU
    
    
    
    
    
    
    
    
    ᵋᴷᴷ
    EFW
    
    
    
    
    
    
    
    
    
20. $ droot run • ࢦఆͨ͠σΟϨΫτϦΛchroot jailͱ࣮ͯ͠ߦ • σόΠεϑΝΠϧͷ࡞੒ (/dev/null, /dev/zeroͳͲ) • ϗετͷ /etc/group, /etc/resolve.confͳͲΛίϐʔ • bind mountͰϗετଆͷ೚ҙͷσΟϨΫτϦΛϚ΢ϯτ • Linux capabilities(7) ͰݖݶΛ཈੍ 
    TVEP
    ESPPU
    SVO
    DQ
    CJOE
    WBSMPH
    SPPU
    ɹɹWBSDPOUBJOFSTBQQ
    
    DPNNBOE
21. chroot(2) • ϓϩηεͷϧʔτσΟϨΫτϦΛมߋ • ϓϩηεͷઈରύεͷ୳ࡧى఺ͷมߋͷΈ • ϓϩηεΛੜ੒ͨ͠Γ͠ͳ͍ • ΧϨϯτσΟϨΫτϦ͸ͦͷ··ͳͷͰcrhootίʔ ϧޙʹchdir(“/“)͢Δ͜ͱ͕ଟ͍ • jail؀ڥ֎ͷϑΝΠϧ΁షΒΕͨγϯϘϦοΫϦϯ Ϋ΁͸ΞΫηεͰ͖ͳ͍
22. BindϚ΢ϯτ • Linux 2.2͔Βಋೖ • σΟϨΫτϦ΍ϑΝΠϧΛଞͷҐஔ΁Ϛ΢ϯτ • chroot jail؀ڥ಺͔ΒϚ΢ϯτઌͷϑΝΠϧ΍σΟϨ ΫτϦ΁ΞΫηεͰ͖Δ • /var/containers/app/var/log ͱ͔ࢀর͢Δͷ໘౗ • mount -o bind /var/log /var/containers/app/var/log • ϗετͷ /var/log Λڞ༗
23. LinuxέʔύϏϦςΟ • chroot(2)͸ಛݖϓϩηεͰͳ͍ͱίʔϧͰ͖ͳ͍ • (ݫີʹ͸CAP_SYS_CHROOT) • ͔͠͠ɺεʔύʔϢʔβͰಈ͔͢ͷ͸ෆ҆ • εʔύʔϢʔβͰಈ͔ͭͭ͠ɺcapabilities(7)Ͱඞཁͳ ݖݶҎ֎Λམͱ͓ͯ͘͠ • CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_NET_BIND_SERVICE ΛڐՄ
24. Problems
25. • Docker (NamespacesΛ࢖ͬͨίϯςφ)΄Ͳͷɹ ϙʔλϏϦςΟ͸ͳ͍ • Dockerίϯςφͷ؀ڥม਺͕Ҿ͖ܧ͕Εͳ͍ • Dockerίϯςφ্ͷ user/group ͕σϓϩΠઌ ϗετʹ͍ͳ͍ ϙʔλϏϦςΟͷ໰୊
26. • ؀ڥม਺͸ϑΝΠϧͱͯ͠ӬଓԽ͞Εͳ͍ͷͰɺ Ұ୴ϑΝΠϧʹอଘ • droot export ͰҰ୴Dockerίϯςφͱͯ͠ىಈ͞ ͔ͤͯΒ env ίϚϯυͷ࣮ߦ݁ՌΛ /.drootenv ͱ ͯ͠อଘ • droot run Ͱ /.drootnenv ΛಡΈͩͯ͠؀ڥม਺Λ ෮ݩ • droot run —env Ͱ؀ڥม਺ͷ௥Ճɾ্ॻ͖Մೳ ؀ڥม਺ͷҾ͖ܧ͗
27. • User Namespaces಺Ͱ/etc/groupͳͲΛΈͯඞཁ ͳuser/grpupΛࣗಈ࡞੒ • ϓϩηεπϦʔߏ଄Λ͔͑ͨ͘ͳ͍ͷͰɺclone(2) Ͱ͸ͳ͘ chroot(2) ޙʹ unshare(2) ͢Δ • clone(2) ͩͱࢠϓϩηεΛੜ੒͢ΔͨΊɺεʔ ύʔόΠβϓϩηεͷ഑ԼͰdroot runͨ͠ͱ͖ ʹɺγάφϧ؅ཧ͕͏·͍͔͘ͳ͍͔΋ user/groupͷࣗಈ࡞੒(ະ࣮૷)
28. • ͜Ε͸͓ͦΒ͘ PID Namespacesͷ࿩ • https://lwn.net/Articles/532748/ • Namespaces௚Լͷϓϩηε͕pid 1ͱͯ͠ৼΔ෣͏ඞཁ͕Ͱ ͯ͘Δ • orphanϓϩηεͷճऩ͢Δඞཁ͕͋Δ
29. ίϯςφ͸ ࣗ෼Ͱ࡞ΕΔ
30. (PMBOH
31. • github.com/docker/docker/pkg • archive, devicemapper, fileutils, mount, symlink… • github.com/opencontainers/runc/libcontainer • Linux Namespaces·ΘΓ • https://github.com/syndtr/gocapability • LinuxέʔύϏϦςΟ • github.com/docker/engine-api • Docker APIΫϥΠΞϯτ ίϯςφπʔϧ޲͚ύοέʔδ
32. • ࣗ࡞ͷίϯςφπʔϧ Drootͷഎܠͱ࣮૷ • Build, Ship, RunΛ࣮ݱ͢Δୈ̏ͷιϑτ΢ΣΞ • droot export, droot deploy, droot run • DockerͰΠϝʔδΛ࡞ͬͯ chroot Ͱ࣮ߦ • ϙʔλϏϦςΟͷ໰୊ͱͦͷղܾ • ίϯςφ͸ࣗ෼Ͱ࡞ΕΔ ·ͱΊ
33. github.com/yuuki/droot