$30 off During Our Annual Pro Sale. View Details »

Droot Internals

Droot Internals

id:y_uuki
第9回 コンテナ型仮想化の情報交換会@福岡

Yuuki Tsubouchi (yuuk1)

April 23, 2016
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Technology

Transcript

  1. %SPPU*OUFSOBMT
    JEZ@VVLJ
    ୈ̕ճίϯςφܕԾ૝Խͷ৘ใަ׵ձ!෱Ԭ

    View Slide

  2. id:y_uuki
    @y_uuk1
    ͸ͯͳ!ژ౎
    ΢ΣϒΦϖϨʔγϣϯΤϯδχΞ

    View Slide

  3. IUUQZVVLJIBUFOBCMPHDPNFOUSZESPPU

    View Slide

  4. TL;DR
    • ιϑτ΢ΣΞґଘ஍ࠈͷղܾͷͨΊʹ
    DockerΛ࢖͍͍ͨ
    • ຊ൪؀ڥͰDockerΛӡ༻͢Δͷ͸ͭΒ͍
    • ʮBuild, Ship, Runʯͱ͍͏ίϯηϓτ͸޷͖
    • DockerΠϝʔδΛS3Λܦ༝ͯ͠഑෍͠ɺ
    chrootͰ࣮ߦ͢Δख๏ͷఏҊ

    View Slide

  5. Ծ૝Խٕज़
    )8Ծ૝Խ 04Ծ૝Խ
    ,7.
    9FO
    ʜ
    γεςϜίϯςφ
    -9$

    ΞϓϦέʔγϣϯ
    ίϯςφ
    %PDLFS

    View Slide

  6. ͳͥ(๻͸)DockerΛ࢖͏ͷ͔
    • ✘ VMΑΓߴ଎
    • ✘ Πϛϡʔλϒϧ
    • ✘ Φʔτεέʔϧ
    • ˚ ϙʔλϏϦςΟ
    • ○ ϓϩάϥϚϒϧͳϗετ؀ڥ
    • ○ ιϑτ΢ΣΞґଘ஍ࠈͷղܾ

    View Slide

  7. ιϑτ΢ΣΞґଘ஍ࠈ
    • ͋Διϑτ΢ΣΞ͸͍͍ͨͯෳ਺ͷιϑτ
    ΢ΣΞʹґଘ͢Δ
    • ґଘઌͷιϑτ΢ΣΞ΋·ͨෳ਺ͷιϑτ
    ΢ΣΞʹґଘ͢Δ
    • ಉ͡؀ڥΛ࠶ݱ͢Δͷ͕೉͍͠
    • BundlerͳͲΛ࢖ͬͯ΋CͷϥΠϒϥϦʹ
    ґଘ͢Δ͜ͱ΋

    View Slide

  8. Docker
    • LinuxͷσΟετϦϏϡʔγϣϯ؀ڥ
    ͝ͱݻΊͯΠϝʔδԽ
    • /lib, /usr/bin, /etcͳͲͥΜͿ
    • Linux NamespacesͰಠཱͨ͠؀ڥΛ
    ࡞ͬͯΠϝʔδΛల։

    View Slide

  9. DockerࠔΔ͜ͱ
    • Docker Engineͷෆ҆ఆ͞
    • ωοτϫʔΫ·ΘΓͷύϑΥʔϚϯεྼԽ
    • ίϯςφͷΰϛ૟আ
    • ίϯςφͷແఀࢭσϓϩΠ
    • ίϯςφͷϩά؅ཧ
    • ίϯςφͷ؂ࢹ
    • ίϯςφͷσόοά
    • Docker Registryͷӡ༻

    View Slide

  10. chroot

    View Slide

  11. chroot ☓ Docker ͷΞΠσΞ
    EPDLFSQVMMNZTRM
    $0/5"*/&3@*% EPDLFSDSFBUFNZTRM

    EPDLFSFYQPSU$0/5"*/&3@*%PNZTRMUBS
    NZTRMUBSΛ.Z42-Λಈ͔͍ͨ͠ϗετ΁ίϐʔ
    ͢Δɻ
    UBSYG[WBSDPOUBJOFSTNZTRMNZTRMUBS
    TVEPDISPPUWBSDPOUBJOFSTNZTRMNZTRME

    View Slide

  12. View Slide

  13. ࠷ۙͷΞϓϦέʔγϣϯσϓϩΠ
    • git pull͕஗͍
    • σϓϩΠαʔό͔Βͷrsync΋஗͍
    • tarball σϓϩΠ
    • ੒Ռ෺ΛS3ͳͲʹ഑ஔ͠ɺσϓϩΠઌͰ s3 cp
    Ͱμ΢ϯϩʔυͯ͠ల։
    • Serf/Consul
    • σϓϩΠαʔό͔ΒͷSSH͕஗͍ͨΊ

    View Slide

  14. Droot

    View Slide

  15. #VJME
    4IJQ
    3VO ESPPUSVO
    EPDLFSCVJME
    ESPPUFYQPSU
    EPDLFS
    EBFNPO
    4UPSBHF
    4

    ESPPUEFQMPZ
    BXTTDQ
    BXTTDQ

    View Slide

  16. %PDLFS %SPPU
    #VJME EPDLFSCVJME EPDLFSCVJME
    3FHJTUSZ
    %PDLFS)VC
    %JTUSJCVUJPO
    ͳΜͰ΋Α͍

    "NB[PO4

    'JMF'PSNBU %PDLFSJNBHF
    ͳΜͰ΋Α͍
    FYUBSH[

    $POUBJOFS
    -JOVY
    /BNFTQBDFT
    DISPPU

    View Slide

  17. $ droot export
    • DockerΠϝʔδͷϑΝΠϧγεςϜΛtarܗࣜͰग़ྗ
    • جຊ͸ docker create && docker export
    • gzip / aws cli ͱͷύΠϓʹΑΓɺtar.gzԽͯ͠S3ʹ഑ஔ
    • ϑΝΠϧγεςϜʹdrootઐ༻ͷ؀ڥม਺ϑΝΠϧ
    (/.drootenv) Λ࢓ࠐΉ
    ESPPUFYQPSUEPDLFSpMFTBQQcH[JQDR
    cBXTTDQTCVDLFUBQQUBSH[

    View Slide

  18. $ droot deploy
    • ඪ४ೖྗ͔ΒtarΞʔΧΠϒΛಡΈࠐΈɺࢦఆ͠
    ͨσΟϨΫτϦʹల։
    • සൟʹߋ৽͞ΕΔίϯςφͷσϓϩΠΛ૝ఆ
    • rsync mode ͱ symlink mode
    BXTTDQTCVDLFUBQQUBSH[c
    HVO[JQDRcESPPUEFQMPZSPPUWBSDPOUBJOFST

    View Slide

  19. symlink ʹΑΔ atomic deploy
    • σϓϩΠࡁΈͷίϯςφ؀
    ڥΛࠩ͠ସ͑Δඞཁ͕͋Δ
    • https://gist.github.com/
    datagrok/3807742#file-
    symlink-replacement-md
    • symlink Λ rename(2)ɹ
    (mv -T) Ͱ੾Γସ͑Δ͜ͱ
    ʹΑΓΞτϛοΫʹσΟϨ
    ΫτϦΛࠩ͠ସ͑Δ

    ᵓᴷᴷBQQBQQENBJO
    ᵓᴷᴷBQQE
    ᵋᴷᴷNBJO
    ᵋᴷᴷCJO
    ᵋᴷᴷCPPU
    ᵋᴷᴷEFW

    ᵋᴷᴷCBDLVQ
    ᵋᴷᴷCJO
    ᵋᴷᴷCPPU
    ᵋᴷᴷEFW

    View Slide

  20. $ droot run
    • ࢦఆͨ͠σΟϨΫτϦΛchroot jailͱ࣮ͯ͠ߦ
    • σόΠεϑΝΠϧͷ࡞੒ (/dev/null, /dev/zeroͳͲ)
    • ϗετͷ /etc/group, /etc/resolve.confͳͲΛίϐʔ
    • bind mountͰϗετଆͷ೚ҙͷσΟϨΫτϦΛϚ΢ϯτ
    • Linux capabilities(7) ͰݖݶΛ཈੍
    TVEPESPPUSVODQCJOEWBSMPHSPPU
    ɹɹWBSDPOUBJOFSTBQQDPNNBOE

    View Slide

  21. chroot(2)
    • ϓϩηεͷϧʔτσΟϨΫτϦΛมߋ
    • ϓϩηεͷઈରύεͷ୳ࡧى఺ͷมߋͷΈ
    • ϓϩηεΛੜ੒ͨ͠Γ͠ͳ͍
    • ΧϨϯτσΟϨΫτϦ͸ͦͷ··ͳͷͰcrhootίʔ
    ϧޙʹchdir(“/“)͢Δ͜ͱ͕ଟ͍
    • jail؀ڥ֎ͷϑΝΠϧ΁షΒΕͨγϯϘϦοΫϦϯ
    Ϋ΁͸ΞΫηεͰ͖ͳ͍

    View Slide

  22. BindϚ΢ϯτ
    • Linux 2.2͔Βಋೖ
    • σΟϨΫτϦ΍ϑΝΠϧΛଞͷҐஔ΁Ϛ΢ϯτ
    • chroot jail؀ڥ಺͔ΒϚ΢ϯτઌͷϑΝΠϧ΍σΟϨ
    ΫτϦ΁ΞΫηεͰ͖Δ
    • /var/containers/app/var/log ͱ͔ࢀর͢Δͷ໘౗
    • mount -o bind /var/log /var/containers/app/var/log
    • ϗετͷ /var/log Λڞ༗

    View Slide

  23. LinuxέʔύϏϦςΟ
    • chroot(2)͸ಛݖϓϩηεͰͳ͍ͱίʔϧͰ͖ͳ͍
    • (ݫີʹ͸CAP_SYS_CHROOT)
    • ͔͠͠ɺεʔύʔϢʔβͰಈ͔͢ͷ͸ෆ҆
    • εʔύʔϢʔβͰಈ͔ͭͭ͠ɺcapabilities(7)Ͱඞཁͳ
    ݖݶҎ֎Λམͱ͓ͯ͘͠
    • CAP_CHOWN, CAP_DAC_OVERRIDE,
    CAP_DAC_READ_SEARCH, CAP_FOWNER,
    CAP_SETGID, CAP_SETUID, CAP_NET_BIND_SERVICE
    ΛڐՄ

    View Slide

  24. Problems

    View Slide

  25. • Docker (NamespacesΛ࢖ͬͨίϯςφ)΄Ͳͷɹ
    ϙʔλϏϦςΟ͸ͳ͍
    • Dockerίϯςφͷ؀ڥม਺͕Ҿ͖ܧ͕Εͳ͍
    • Dockerίϯςφ্ͷ user/group ͕σϓϩΠઌ
    ϗετʹ͍ͳ͍
    ϙʔλϏϦςΟͷ໰୊

    View Slide

  26. • ؀ڥม਺͸ϑΝΠϧͱͯ͠ӬଓԽ͞Εͳ͍ͷͰɺ
    Ұ୴ϑΝΠϧʹอଘ
    • droot export ͰҰ୴Dockerίϯςφͱͯ͠ىಈ͞
    ͔ͤͯΒ env ίϚϯυͷ࣮ߦ݁ՌΛ /.drootenv ͱ
    ͯ͠อଘ
    • droot run Ͱ /.drootnenv ΛಡΈͩͯ͠؀ڥม਺Λ
    ෮ݩ
    • droot run —env Ͱ؀ڥม਺ͷ௥Ճɾ্ॻ͖Մೳ
    ؀ڥม਺ͷҾ͖ܧ͗

    View Slide

  27. • User Namespaces಺Ͱ/etc/groupͳͲΛΈͯඞཁ
    ͳuser/grpupΛࣗಈ࡞੒
    • ϓϩηεπϦʔߏ଄Λ͔͑ͨ͘ͳ͍ͷͰɺclone(2)
    Ͱ͸ͳ͘ chroot(2) ޙʹ unshare(2) ͢Δ
    • clone(2) ͩͱࢠϓϩηεΛੜ੒͢ΔͨΊɺεʔ
    ύʔόΠβϓϩηεͷ഑ԼͰdroot runͨ͠ͱ͖
    ʹɺγάφϧ؅ཧ͕͏·͍͔͘ͳ͍͔΋
    user/groupͷࣗಈ࡞੒(ະ࣮૷)

    View Slide

  28. • ͜Ε͸͓ͦΒ͘ PID Namespacesͷ࿩
    • https://lwn.net/Articles/532748/
    • Namespaces௚Լͷϓϩηε͕pid 1ͱͯ͠ৼΔ෣͏ඞཁ͕Ͱ
    ͯ͘Δ
    • orphanϓϩηεͷճऩ͢Δඞཁ͕͋Δ

    View Slide

  29. ίϯςφ͸
    ࣗ෼Ͱ࡞ΕΔ

    View Slide

  30. (PMBOH

    View Slide

  31. • github.com/docker/docker/pkg
    • archive, devicemapper, fileutils, mount, symlink…
    • github.com/opencontainers/runc/libcontainer
    • Linux Namespaces·ΘΓ
    • https://github.com/syndtr/gocapability
    • LinuxέʔύϏϦςΟ
    • github.com/docker/engine-api
    • Docker APIΫϥΠΞϯτ
    ίϯςφπʔϧ޲͚ύοέʔδ

    View Slide

  32. • ࣗ࡞ͷίϯςφπʔϧ Drootͷഎܠͱ࣮૷
    • Build, Ship, RunΛ࣮ݱ͢Δୈ̏ͷιϑτ΢ΣΞ
    • droot export, droot deploy, droot run
    • DockerͰΠϝʔδΛ࡞ͬͯ chroot Ͱ࣮ߦ
    • ϙʔλϏϦςΟͷ໰୊ͱͦͷղܾ
    • ίϯςφ͸ࣗ෼Ͱ࡞ΕΔ
    ·ͱΊ

    View Slide

  33. github.com/yuuki/droot

    View Slide