Droot Internals

Droot Internals

id:y_uuki
第9回 コンテナ型仮想化の情報交換会@福岡

A658ec7f1badf73819dfa501165016c1?s=128

Yuuki Tsubouchi (yuuk1)

April 23, 2016
Tweet

Transcript

  1. %SPPU*OUFSOBMT JEZ@VVLJ ୈ̕ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ!෱Ԭ

  2. id:y_uuki @y_uuk1 ͸ͯͳ!ژ౎ ΢ΣϒΦϖϨʔγϣϯΤϯδχΞ

  3. IUUQZVVLJIBUFOBCMPHDPNFOUSZESPPU

  4. TL;DR • ιϑτ΢ΣΞґଘ஍ࠈͷղܾͷͨΊʹ DockerΛ࢖͍͍ͨ • ຊ൪؀ڥͰDockerΛӡ༻͢Δͷ͸ͭΒ͍ • ʮBuild, Ship, Runʯͱ͍͏ίϯηϓτ͸޷͖

    • DockerΠϝʔδΛS3Λܦ༝ͯ͠഑෍͠ɺ chrootͰ࣮ߦ͢Δख๏ͷఏҊ
  5. Ծ૝Խٕज़ )8Ծ૝Խ 04Ծ૝Խ ,7. 9FO ʜ γεςϜίϯςφ -9$ ΞϓϦέʔγϣϯ ίϯςφ

    %PDLFS
  6. ͳͥ(๻͸)DockerΛ࢖͏ͷ͔ • ✘ VMΑΓߴ଎ • ✘ Πϛϡʔλϒϧ • ✘ Φʔτεέʔϧ

    • ˚ ϙʔλϏϦςΟ • ◦ ϓϩάϥϚϒϧͳϗετ؀ڥ • ◦ ιϑτ΢ΣΞґଘ஍ࠈͷղܾ
  7. ιϑτ΢ΣΞґଘ஍ࠈ • ͋Διϑτ΢ΣΞ͸͍͍ͨͯෳ਺ͷιϑτ ΢ΣΞʹґଘ͢Δ • ґଘઌͷιϑτ΢ΣΞ΋·ͨෳ਺ͷιϑτ ΢ΣΞʹґଘ͢Δ • ಉ͡؀ڥΛ࠶ݱ͢Δͷ͕೉͍͠ •

    BundlerͳͲΛ࢖ͬͯ΋CͷϥΠϒϥϦʹ ґଘ͢Δ͜ͱ΋
  8. Docker • LinuxͷσΟετϦϏϡʔγϣϯ؀ڥ ͝ͱݻΊͯΠϝʔδԽ • /lib, /usr/bin, /etcͳͲͥΜͿ • Linux

    NamespacesͰಠཱͨ͠؀ڥΛ ࡞ͬͯΠϝʔδΛల։
  9. DockerࠔΔ͜ͱ • Docker Engineͷෆ҆ఆ͞ • ωοτϫʔΫ·ΘΓͷύϑΥʔϚϯεྼԽ • ίϯςφͷΰϛ૟আ • ίϯςφͷແఀࢭσϓϩΠ

    • ίϯςφͷϩά؅ཧ • ίϯςφͷ؂ࢹ • ίϯςφͷσόοά • Docker Registryͷӡ༻
  10. chroot

  11. chroot ☓ Docker ͷΞΠσΞ EPDLFSQVMMNZTRM $0/5"*/&3@*% EPDLFSDSFBUFNZTRM  EPDLFSFYQPSU$0/5"*/&3@*%PNZTRMUBS NZTRMUBSΛ.Z42-Λಈ͔͍ͨ͠ϗετ΁ίϐʔ

    ͢Δɻ UBSYG[WBSDPOUBJOFSTNZTRMNZTRMUBS TVEPDISPPUWBSDPOUBJOFSTNZTRMNZTRME
  12. None
  13. ࠷ۙͷΞϓϦέʔγϣϯσϓϩΠ • git pull͕஗͍ • σϓϩΠαʔό͔Βͷrsync΋஗͍ • tarball σϓϩΠ •

    ੒Ռ෺ΛS3ͳͲʹ഑ஔ͠ɺσϓϩΠઌͰ s3 cp Ͱμ΢ϯϩʔυͯ͠ల։ • Serf/Consul • σϓϩΠαʔό͔ΒͷSSH͕஗͍ͨΊ
  14. Droot

  15. #VJME 4IJQ 3VO ESPPUSVO EPDLFSCVJME ESPPUFYQPSU EPDLFS EBFNPO 4UPSBHF 4

    ESPPUEFQMPZ BXTTDQ BXTTDQ
  16. %PDLFS %SPPU #VJME EPDLFSCVJME EPDLFSCVJME 3FHJTUSZ %PDLFS)VC %JTUSJCVUJPO ͳΜͰ΋Α͍
 "NB[PO4

    'JMF'PSNBU %PDLFSJNBHF ͳΜͰ΋Α͍ FYUBSH[ $POUBJOFS -JOVY /BNFTQBDFT DISPPU
  17. $ droot export • DockerΠϝʔδͷϑΝΠϧγεςϜΛtarܗࣜͰग़ྗ • جຊ͸ docker create &&

    docker export • gzip / aws cli ͱͷύΠϓʹΑΓɺtar.gzԽͯ͠S3ʹ഑ஔ • ϑΝΠϧγεςϜʹdrootઐ༻ͷ؀ڥม਺ϑΝΠϧ (/.drootenv) Λ࢓ࠐΉ ESPPUFYQPSUEPDLFSpMFTBQQcH[JQDR cBXTTDQTCVDLFUBQQUBSH[
  18. $ droot deploy • ඪ४ೖྗ͔ΒtarΞʔΧΠϒΛಡΈࠐΈɺࢦఆ͠ ͨσΟϨΫτϦʹల։ • සൟʹߋ৽͞ΕΔίϯςφͷσϓϩΠΛ૝ఆ • rsync

    mode ͱ symlink mode BXTTDQTCVDLFUBQQUBSH[c HVO[JQDRcESPPUEFQMPZSPPUWBSDPOUBJOFST
  19. symlink ʹΑΔ atomic deploy • σϓϩΠࡁΈͷίϯςφ؀ ڥΛࠩ͠ସ͑Δඞཁ͕͋Δ • https://gist.github.com/ datagrok/3807742#file-

    symlink-replacement-md • symlink Λ rename(2)ɹ (mv -T) Ͱ੾Γସ͑Δ͜ͱ ʹΑΓΞτϛοΫʹσΟϨ ΫτϦΛࠩ͠ସ͑Δ  ᵓᴷᴷBQQBQQENBJO ᵓᴷᴷBQQE ᵋᴷᴷNBJO ᵋᴷᴷCJO ᵋᴷᴷCPPU ᵋᴷᴷEFW  ᵋᴷᴷCBDLVQ ᵋᴷᴷCJO ᵋᴷᴷCPPU ᵋᴷᴷEFW 
  20. $ droot run • ࢦఆͨ͠σΟϨΫτϦΛchroot jailͱ࣮ͯ͠ߦ • σόΠεϑΝΠϧͷ࡞੒ (/dev/null, /dev/zeroͳͲ)

    • ϗετͷ /etc/group, /etc/resolve.confͳͲΛίϐʔ • bind mountͰϗετଆͷ೚ҙͷσΟϨΫτϦΛϚ΢ϯτ • Linux capabilities(7) ͰݖݶΛ཈੍ TVEPESPPUSVODQCJOEWBSMPHSPPU ɹɹWBSDPOUBJOFSTBQQDPNNBOE
  21. chroot(2) • ϓϩηεͷϧʔτσΟϨΫτϦΛมߋ • ϓϩηεͷઈରύεͷ୳ࡧى఺ͷมߋͷΈ • ϓϩηεΛੜ੒ͨ͠Γ͠ͳ͍ • ΧϨϯτσΟϨΫτϦ͸ͦͷ··ͳͷͰcrhootίʔ ϧޙʹchdir(“/“)͢Δ͜ͱ͕ଟ͍

    • jail؀ڥ֎ͷϑΝΠϧ΁షΒΕͨγϯϘϦοΫϦϯ Ϋ΁͸ΞΫηεͰ͖ͳ͍
  22. BindϚ΢ϯτ • Linux 2.2͔Βಋೖ • σΟϨΫτϦ΍ϑΝΠϧΛଞͷҐஔ΁Ϛ΢ϯτ • chroot jail؀ڥ಺͔ΒϚ΢ϯτઌͷϑΝΠϧ΍σΟϨ ΫτϦ΁ΞΫηεͰ͖Δ

    • /var/containers/app/var/log ͱ͔ࢀর͢Δͷ໘౗ • mount -o bind /var/log /var/containers/app/var/log • ϗετͷ /var/log Λڞ༗
  23. LinuxέʔύϏϦςΟ • chroot(2)͸ಛݖϓϩηεͰͳ͍ͱίʔϧͰ͖ͳ͍ • (ݫີʹ͸CAP_SYS_CHROOT) • ͔͠͠ɺεʔύʔϢʔβͰಈ͔͢ͷ͸ෆ҆ • εʔύʔϢʔβͰಈ͔ͭͭ͠ɺcapabilities(7)Ͱඞཁͳ ݖݶҎ֎Λམͱ͓ͯ͘͠

    • CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_NET_BIND_SERVICE ΛڐՄ
  24. Problems

  25. • Docker (NamespacesΛ࢖ͬͨίϯςφ)΄Ͳͷɹ ϙʔλϏϦςΟ͸ͳ͍ • Dockerίϯςφͷ؀ڥม਺͕Ҿ͖ܧ͕Εͳ͍ • Dockerίϯςφ্ͷ user/group ͕σϓϩΠઌ

    ϗετʹ͍ͳ͍ ϙʔλϏϦςΟͷ໰୊
  26. • ؀ڥม਺͸ϑΝΠϧͱͯ͠ӬଓԽ͞Εͳ͍ͷͰɺ Ұ୴ϑΝΠϧʹอଘ • droot export ͰҰ୴Dockerίϯςφͱͯ͠ىಈ͞ ͔ͤͯΒ env ίϚϯυͷ࣮ߦ݁ՌΛ

    /.drootenv ͱ ͯ͠อଘ • droot run Ͱ /.drootnenv ΛಡΈͩͯ͠؀ڥม਺Λ ෮ݩ • droot run —env Ͱ؀ڥม਺ͷ௥Ճɾ্ॻ͖Մೳ ؀ڥม਺ͷҾ͖ܧ͗
  27. • User Namespaces಺Ͱ/etc/groupͳͲΛΈͯඞཁ ͳuser/grpupΛࣗಈ࡞੒ • ϓϩηεπϦʔߏ଄Λ͔͑ͨ͘ͳ͍ͷͰɺclone(2) Ͱ͸ͳ͘ chroot(2) ޙʹ unshare(2)

    ͢Δ • clone(2) ͩͱࢠϓϩηεΛੜ੒͢ΔͨΊɺεʔ ύʔόΠβϓϩηεͷ഑ԼͰdroot runͨ͠ͱ͖ ʹɺγάφϧ؅ཧ͕͏·͍͔͘ͳ͍͔΋ user/groupͷࣗಈ࡞੒(ະ࣮૷)
  28. • ͜Ε͸͓ͦΒ͘ PID Namespacesͷ࿩ • https://lwn.net/Articles/532748/ • Namespaces௚Լͷϓϩηε͕pid 1ͱͯ͠ৼΔ෣͏ඞཁ͕Ͱ ͯ͘Δ

    • orphanϓϩηεͷճऩ͢Δඞཁ͕͋Δ
  29. ίϯςφ͸ ࣗ෼Ͱ࡞ΕΔ

  30. (PMBOH

  31. • github.com/docker/docker/pkg • archive, devicemapper, fileutils, mount, symlink… • github.com/opencontainers/runc/libcontainer

    • Linux Namespaces·ΘΓ • https://github.com/syndtr/gocapability • LinuxέʔύϏϦςΟ • github.com/docker/engine-api • Docker APIΫϥΠΞϯτ ίϯςφπʔϧ޲͚ύοέʔδ
  32. • ࣗ࡞ͷίϯςφπʔϧ Drootͷഎܠͱ࣮૷ • Build, Ship, RunΛ࣮ݱ͢Δୈ̏ͷιϑτ΢ΣΞ • droot export,

    droot deploy, droot run • DockerͰΠϝʔδΛ࡞ͬͯ chroot Ͱ࣮ߦ • ϙʔλϏϦςΟͷ໰୊ͱͦͷղܾ • ίϯςφ͸ࣗ෼Ͱ࡞ΕΔ ·ͱΊ
  33. github.com/yuuki/droot