$30 off During Our Annual Pro Sale. View Details »

Better docker image+

orisano
December 05, 2018

Better docker image+

orisano

December 05, 2018
Tweet

More Decks by orisano

Other Decks in Technology

Transcript

  1. Better Docker Image+
    Bonfire Backend #2 #yjbonfire
    @orisano

    View Slide

  2. ྑ͍Docker Imageͱ͸Կ͔

    View Slide

  3. ݟͯΘ͔Γ΍͍͢
    खݩͰߴ଎ʹbuild͞ΕΔ
    CI্Ͱߴ଎ʹbuild͞ΕΔ
    ߴ଎ʹdeploy͞ΕΔ

    View Slide

  4. ࠓ೔࿩͍ͨ͜͠ͱ

    View Slide

  5. ͲͷΑ͏ʹ଎͘͢Δ͔
    ͲͷΑ͏ʹখ͘͢͞Δ͔

    View Slide

  6. ͲͷΑ͏ʹ଎͘͢Δ͔
    ͲͷΑ͏ʹখ͘͢͞Δ͔

    View Slide

  7. ͲͷΑ͏ʹ଎͘͢Δ͔
    • ΠϝʔδΛখ͘͢͞Δ
    • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ
    • cacheΛޮ͔ͤΔ
    • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ
    • ඞཁͳ͍εςʔδΛbuild͠ͳ͍

    View Slide

  8. ͲͷΑ͏ʹ଎͘͢Δ͔
    • ΠϝʔδΛখ͘͢͞Δ
    • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ
    • cacheΛޮ͔ͤΔ
    • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ
    • ඞཁͳ͍εςʔδΛbuild͠ͳ͍

    View Slide

  9. imageΛখ͘͢͞Δͷ͸
    docker push͢Δͱ͖ͷ଎౓
    docker pull͢Δͱ͖ͷ଎౓
    ͷߴ଎Խʹͭͳ͕Δ

    View Slide

  10. Ͳͷ༷ʹখ͘͢͞Δ͔͸
    ͋ͱͰ

    View Slide

  11. ͲͷΑ͏ʹ଎͘͢Δ͔
    • ΠϝʔδΛখ͘͢͞Δ
    • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ
    • cacheΛޮ͔ͤΔ
    • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ
    • ඞཁͳ͍εςʔδΛbuild͠ͳ͍

    View Slide

  12. ίϚϯυͦͷ΋ͷΛ଎͘͢Δ

    View Slide

  13. URLʹର͢ΔADDΛ࢖Θͳ͍

    View Slide

  14. URLʹର͢ΔADD͸
    جຊతʹμ΢ϯϩʔυ͢Δ

    View Slide

  15. ஗͍

    View Slide

  16. ΞΫηε͍ͯ͠Δઌͷ
    ίϯςϯπ͕ႈ౳ͳΒ
    wget + gzip + tarͰे෼

    View Slide

  17. `RUN wget`ʹ͢Δ͜ͱͰ
    cache͕ޮ͘

    View Slide

  18. ႈ౳Ͱͳ͍Ϧιʔεʹ
    ґଘ͢ΔͷΛۃྗ΍ΊΔ

    View Slide

  19. build context
    ͷసૹྔΛߟ͑Δ

    View Slide

  20. ϞϊϨϙʹ͢Δͱ
    build contextେ͖͘ͳΓ͕ͪ

    View Slide

  21. ϞϊϨϙͷ৔߹͸
    .dockerignore
    Λࣗಈੜ੒͠Α͏

    View Slide

  22. github.com/orisano/dignore
    • ࢦఆ͞ΕͨdirectoryҎ֎ͷdirectoryΛignore
    • ࢦఆ͞Εͨdirectoryʹdockerignore͕͋Ε͹
    ల։
    • ͜Ε͚ͩʂ

    View Slide

  23. buildͷલ޻ఔͰ
    buildʹඞཁͳ͍αʔϏε͸
    Ignore͠Α͏

    View Slide

  24. buildkitΛ࢖͍ͬͯΔͱ
    ࠩ෼సૹͯ͘͠ΕͨΓ͢Δ

    View Slide

  25. COPYΛҙࣝͨ͠
    directoryߏ଄

    View Slide

  26. COPYͷҾ਺͕
    directoryͷ৔߹͸
    ର৅ͷdirectoryʹ
    த਎Λશ෦ίϐʔͯ͠͠·͏

    View Slide

  27. ಛఆͷσΟϨΫτϦ͚ͩ
    ίϐʔͨ͘͠ͳ͍
    ͱ͍͏͕೉͍͠

    View Slide

  28. dockerignore
    Ͱআ֎͢Ε͹ྑ͘ͳ͍ʁ

    View Slide

  29. ͍͍͑

    View Slide

  30. vendorͳͲΛ௚઀؅ཧͯ͠
    build࣌ʹdownload͠ͳ͍
    ৔߹ʹຊ౰ʹਏ͍

    View Slide

  31. 2ճҎ্ॏ͍directoryΛ
    COPYͨ͘͠ͳ͍

    View Slide

  32. ͦ͏ͳΒͳ͍ͨΊͷ
    directoryߏ଄ʹ͓ͯ͜͠͏

    View Slide

  33. ਓ͕ؒॻ͍ͨ΋ͷ͕
    ೖ͍ͬͯΔdirectory
    ֎෦ʹґଘ͍ͯ͠Δ΋ͷ
    ࣗಈੜ੒෺͕ೖ͍ͬͯΔ
    directoryΛ෼཭͠Α͏

    View Slide

  34. buildkitͩͱࠩ෼సૹ

    View Slide

  35. ͍ͣΕʹͤΑ
    ෼཭͓͍ͯͨ͠ํ͕
    cacheʹ༏͍͠

    View Slide

  36. RUNͷத਎Λ଎͘͢Δ

    View Slide

  37. GitHub͔Β࣮ߦϑΝΠϧΛ
    curl or wgetͰऔಘ͢Δ
    ͕஗͍

    View Slide

  38. Ͳ͏ʹ͔଎͘Ͱ͖ͳ͍͔
    curl -vvvΛ͏ͬͯΈͨ

    View Slide

  39. GitHub Release͸S3Ͱ͋Δ
    S3͸Accept-Ranges: bytes
    ͱ͍͏͜ͱ͕Θ͔ͬͨ

    View Slide

  40. Accept-Ranges: bytes
    ͬͯͳΜͩΖ͏

    View Slide

  41. RFC7233, Range Requests
    ൣғΛࢦఆͯ͠
    downloadͰ͖Δ

    View Slide

  42. ͭ·Γ
    ฒྻμ΢ϯϩʔυ͕Մೳ

    View Slide

  43. github.com/orisano/rget

    View Slide

  44. View Slide

  45. ؀ڥʹΑΓ·͕͢
    5min -> 2min

    View Slide

  46. Dockerfileͷ޻෉Ҏ֎ʹ΋
    ଎͘͢ΔΞϓϩʔν͸͋Δ

    View Slide

  47. ͲͷΑ͏ʹ଎͘͢Δ͔
    • ΠϝʔδΛখ͘͢͞Δ
    • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ
    • cacheΛޮ͔ͤΔ
    • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ
    • ඞཁͳ͍εςʔδΛbuild͠ͳ͍

    View Slide

  48. cacheΛޮ͔ͤΔ

    View Slide

  49. docker build͢ΔϚγϯ͕
    ಉҰͷ৔߹
    ಉҰͰͳ͍৔߹

    View Slide

  50. docker build͢ΔϚγϯ͕
    ಉҰͷ৔߹
    ಉҰͰͳ͍৔߹

    View Slide

  51. cacheͷ࢓૊ΈΛཧղ͢Δ

    View Slide

  52. RUN͸
    จࣈྻ͕มΘΒͳ͍ݶΓ
    جຊతʹcache͞ΕΔ

    View Slide

  53. COPY, ADD͢ΔϑΝΠϧͷ
    ಺༰͕มΘͬͨ৔߹
    Ҏ߱ͷRUNͷcache͕ഁغ

    View Slide

  54. ͳͷͰ
    lockfileͳͲΛઌʹίϐʔͯ͠
    install͚ͩͯ͠cacheͤ͞Δ

    View Slide

  55. View Slide

  56. View Slide

  57. docker build͢ΔϚγϯ͕
    ಉҰͷ৔߹
    ಉҰͰͳ͍৔߹

    View Slide

  58. CI্Ͱbuild͢Δͱ͖ͳͲ
    cache͕ͳ͍͜ͱ͕ଟ͍

    View Slide

  59. docker save & load
    or
    docker pull
    docker build —cache-from
    Λ࢖͏

    View Slide

  60. image͕େ͖͍/layer͕ଟ͍
    ৔߹
    buildͷ΄͏͕ૣ͍͜ͱ΋͋Δ

    View Slide

  61. image͕େ͖͍/layer͕ଟ͍
    ৔߹
    buildͷ΄͏͕ૣ͍͜ͱ΋͋Δ
    ܭଌܾͯ͠ΊΔ

    View Slide

  62. ͲͷΑ͏ʹ଎͘͢Δ͔
    • ΠϝʔδΛখ͘͢͞Δ
    • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ
    • cacheΛޮ͔ͤΔ
    • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ
    • ඞཁͳ͍εςʔδΛbuild͠ͳ͍

    View Slide

  63. ґଘͷͳ͍εςʔδΛ
    ฒྻͰ࣮ߦ͢Δ

    View Slide

  64. buildkitΛ࢖͍ͬͯͩ͘͞ʂ
    github.com/moby/buildkit
    export DOCKER_BUILDKIT=1

    View Slide

  65. ͲͷΑ͏ʹ଎͘͢Δ͔
    • ΠϝʔδΛখ͘͢͞Δ
    • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ
    • cacheΛޮ͔ͤΔ
    • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ
    • ඞཁͳ͍εςʔδΛbuild͠ͳ͍

    View Slide

  66. ඞཁͳ͍εςʔδΛ
    build͠ͳ͍

    View Slide

  67. —targetΛ࢖͑͹Α͍ͷͰ͸ʁ

    View Slide

  68. —target͸
    ࢦఆͨ͠εςʔδҎલΛ
    શ෦build͢Δ

    View Slide

  69. multi stage buildΛ
    ౿ΈࠐΜͩ࢖͍ํ͍ͯ͠Δͱ
    ૺ۰͕ͪ͠

    View Slide

  70. multi stage buildͷ
    ౿ΈࠐΜͩ࢖͍ํʹ͍ͭͯ͸

    View Slide

  71. https://speakerdeck.com/orisano/multi-stage-builds-patterns-and-practice

    View Slide

  72. ͓ͦΒ͘buildkitͰͰ͖Δ͕
    buildkit͕࢖͑ͳ͍৔߹ʹ

    View Slide

  73. github.com/orisano/targd
    • DockerfileͷASTΛऔಘ
    • ࢦఆ͞ΕͨεςʔδͷґଘεςʔδΛநग़
    • ґଘεςʔδ͚ͩͷDockerfileΛग़ྗ

    View Slide

  74. github.com/orisano/targd

    View Slide

  75. ͲͷΑ͏ʹ଎͘͢Δ͔
    ͲͷΑ͏ʹখ͘͢͞Δ͔

    View Slide

  76. ͲͷΑ͏ʹখ͘͢͞Δ͔
    • multi stage buildΛ࢖͏
    • RUNΛ·ͱΊΔ(?)
    • ͳͥେ͖͍͔Λ஌Δ

    View Slide

  77. ͲͷΑ͏ʹখ͘͢͞Δ͔
    • multi stage buildΛ࢖͏
    • RUNΛ·ͱΊΔ(?)
    • ͳͥେ͖͍͔Λ஌Δ

    View Slide

  78. multi stage buildΛ࢖͏

    View Slide

  79. multi stage buildͰ
    ͋Γ͕ͪͳٙ໰

    View Slide

  80. ࠷ऴతͳΠϝʔδ͕
    খ͘͞ͳΔ͔Β
    ͦΕ·Ͱͷεςʔδ͸
    ࠷దԽ͠ͳͯ͘ྑ͍ʁ

    View Slide

  81. View Slide

  82. ݸਓతʹ͸No

    View Slide

  83. moby/issues/34715
    —cache-formͩͱ
    multi stage buildͷ
    લஈͷεςʔδͷcache͕
    ޮ͔ͳ͍

    View Slide

  84. ࠷ऴεςʔδ͔͠
    pushͯ͠ͳ͍͔Β౰ͨΓલ

    View Slide

  85. CI্Ͱͷbuild͸
    —cache-fromΛ࢖͏
    ࠷ऴεςʔδ͚ͩcache͢Δ
    ҙຯ͸ͳ͍

    View Slide

  86. multi stage build࣌ʹ
    cacheΛޮ͔͍ͤͨ৔߹͸
    લͷεςʔδ΋
    ໌ࣔతʹpush͢Δ͔͠ͳ͍

    View Slide

  87. ͜ΕΛखಈͰ΍Δͱ
    cache-from஍ࠈʹͳΔ

    View Slide

  88. github.com/orisano/castage
    • DockerfileͷASTΛऔಘ
    • εςʔδҰཡΛऔಘ
    • docker pullΛੜ੒
    • docker buildΛcache-from෇͖Ͱੜ੒

    View Slide

  89. ݁ہpush͢ΔͷͰ
    push/pullͷίετ͕͔͔Δ
    ͢΂ͯͷεςʔδΛฏ౳ʹ
    খ͘͢͞΂͖
    (ݸਓͷݟղͰ͢)

    View Slide

  90. ͲͷΑ͏ʹখ͘͢͞Δ͔
    • multi stage buildΛ࢖͏
    • RUNΛ·ͱΊΔ(?)
    • ͳͥେ͖͍͔Λ஌Δ

    View Slide

  91. RUNΛ·ͱΊΔ(?)

    View Slide

  92. Ͳ͔ͬʔ;͍͊Δ͸
    1ͭͷRUNʹ
    શ෦ॻ͘ͱྑ͍Β͍͠(?)

    View Slide

  93. github.com/orisano/minid
    • DockerfileͷASTΛऔಘ
    • ࿈ଓ͢ΔRUN, COPY, ADDΛ࿈݁͢Δ
    • ݁ՌͷDockerfileΛग़ྗ

    View Slide

  94. github.com/orisano/minid

    View Slide

  95. αΠζ͕খ͘͞ͳͬͨ

    View Slide

  96. Ͱ΋ͳΜͰʁ

    View Slide

  97. ϨΠϠʔͷΦʔόʔϔου͕
    ݮΔ͔Βখ͘͞ͳΔʁ

    View Slide

  98. ͍͍͑

    View Slide

  99. ·ͣ
    Ͳ͏͍͏ܗͰอଘ͞ΕͯΔ͔
    Λ஌Δ

    View Slide

  100. moby/image/spec/v1.md

    View Slide

  101. AUFS

    View Slide

  102. http://docs.docker.jp/engine/userguide/storagedriver/aufs-driver.html

    View Slide

  103. ࡟আ͸
    whiteoutϑΝΠϧͷ௥Ճ
    Ҡಈ͸
    opaqueϑΝΠϧͷ௥Ճ
    ʴ
    ҠಈઌͷϑΝΠϧࠩ෼

    View Slide

  104. Ұ౓Ͱ΋
    RUN,COPY,ADDΛލ͙ͱ
    imageʹ͸࢒ͬͯ͠·͏

    View Slide

  105. ػցతʹͰ΋
    ҰͭͷRUNʹ·ͱΊΔ͜ͱͰ
    ༨ܭͳ΋ͷΛ࡟ݮͰ͖Δ

    View Slide

  106. 1ͭͷϨΠϠʔʹ
    ·ͱΊΔ͜ͱ͸
    ຊ౰ʹਖ਼͍͠ͷ͔

    View Slide

  107. ϨΠϠʔΛผ͚Δ͜ͱʹΑΓ
    ฒྻμ΢ϯϩʔυͷԸܙ
    cacheͷ༗ޮ׆༻

    View Slide

  108. ϨΠϠʔΛผ͚Δ͜ͱʹΑΓ
    ฒྻμ΢ϯϩʔυͷԸܙ
    cacheͷ༗ޮ׆༻
    ܭଌܾͯ͠ΊΔ

    View Slide

  109. ͲͷΑ͏ʹখ͘͢͞Δ͔
    • multi stage buildΛ࢖͏
    • RUNΛ·ͱΊΔ(?)
    • ͳͥେ͖͍͔Λ஌Δ

    View Slide

  110. ͳͥେ͖͍͔Λ஌Δ

    View Slide

  111. ͍͔ͳΔνϡʔχϯάͰ΋
    ܭଌͤͣʹ΍ͬͯ͸͍͚ͳ͍

    View Slide

  112. docker history

    View Slide

  113. docker history

    View Slide

  114. ͲͷϨΠϠʔ͕
    େ͖͍͔͸Θ͔Δ͚Ͳ
    ͳͥେ͖͍͔͸෼͔ΓͮΒ͍

    View Slide

  115. ҙ֎ʹίϚϯυ͕
    Ͳ͏͍͏ϑΝΠϧΛ࡞Δͷ͔
    ஌Βͳ͍

    View Slide

  116. github.com/orisano/dlayer
    • docker saveͰಘΒΕͨtarΛղੳ
    • layerͱcmdͷରԠΛͱΔ
    • layerͷࠩ෼ΛϑΝΠϧαΠζ͕େ͖͍ॱʹද

    View Slide

  117. github.com/orisano/dlayer

    View Slide

  118. ࣮ࡍʹdlayerΛ࢖ͬͯ
    golangެࣜalpineΠϝʔδͷ
    αΠζΛݮΒͯ͠Έͨ

    View Slide

  119. docker-library/golang/pull/232

    View Slide

  120. View Slide

  121. golang:1.11͔Βtoolchainͷ
    αΠζ͕Ͱ͔͘ͳͬͨ
    SSAͰͷ࠷దԽ͕૿͔͑ͨΒʁ

    View Slide

  122. 253MB

    View Slide

  123. 305MB

    View Slide

  124. github.com/wagoodman/dive

    View Slide

  125. ࡞ͬͨπʔϧ
    • github.com/orisano/rget
    • github.com/orisano/targd
    • github.com/orisano/minid
    • github.com/orisano/dlayer
    • ྑ͍ͱࢥͬͨΒελʔ͍͚ͯͨͩ͠ΔͱྭΈʹͳΓ·͢

    View Slide

  126. ࡞ͬͨπʔϧ
    • github.com/orisano/dignore
    • github.com/orisano/castage
    • ྑ͍ͱࢥͬͨΒελʔ͍͚ͯͨͩ͠ΔͱྭΈʹͳΓ·͢

    View Slide

  127. ·ͱΊ
    • ίϚϯυΛ଎͘Ͱ͖ͳ͍͔ߟ͑Δ
    • cacheΛཧղͯ͠༗ޮʹ׆͔͢ (CI or ϩʔΧϧ)
    • multi stage buildΛ࢖͏ (ॏ͍stageΛcache͢Δ)
    • layerΛখ͘͢͞ΔͨΊʹ෼ੳ͢Δ
    • buildkitΛ࢖͏!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    View Slide

  128. rget͜΅Ε࿩

    View Slide

  129. github.com/orisano/rget

    View Slide

  130. Ͱ΋
    ͜ΕͰຊ౰ʹ͍͍ΜͩΖ͏͔

    View Slide

  131. rgetͷbinaryΛdownload
    ͢ΔͷͰ͸ෛ͚ͨؾ͕͢Δ()

    View Slide

  132. shell͚ͩͰ
    ࣮ݱͰ͖ͳ͍ͩΖ͏͔

    View Slide

  133. ϙʔλϒϧੑ͸͍Βͳ͍ͷͰ
    ओઓ৔ͷalpineʹߜΔ

    View Slide

  134. alpine(busybox)͸
    xargs͕ೖ͍ͬͯΔʂʂʂ

    View Slide

  135. xargsΛ࢖͏͜ͱͰ
    ฒྻԽ
    ಉ࣌઀ଓ਺੍ݶ
    ͕࣮ݱͰ͖Δ

    View Slide

  136. GitHub Release͸
    ؆୯ʹHEAD͕Ͱ͖ͳ͍ͷͰ
    GETͰbodyΛແࢹͯ͠
    Content-LengthΛऔಘͰ͖Δ

    View Slide

  137. seqͰ
    chunkͷrangeΛੜ੒Ͱ͖Δ

    View Slide

  138. wget͸206 Partial Content
    ͕͏·͘ѻ͑ͳ͍ͷͰμϝ

    View Slide

  139. ࢓ํͳ͍ͷͰ
    curl -RͰdownload

    View Slide

  140. ࠷ޙʹcatͯ͠chunkΛ݁߹

    View Slide

  141. shellͰ΋࣮ݱͰ͖Δʂ

    View Slide

  142. ͔͠͠
    curl͸alpineඪ४Ͱ͸ͳ͍

    View Slide

  143. apk add —no-cache curl
    ͸ෛ͚ͨؾ͕͢Δ()

    View Slide

  144. ͔͠͠
    wget͸206͕ॲཧͰ͖ͳ͍

    View Slide

  145. ఘΊΒΕͳ͍ͷͰ
    busyboxͷwgetͷ
    ιʔείʔυΛಡΉ

    View Slide

  146. busybox/networking/wget.c

    View Slide

  147. busybox/networking/wget.c

    View Slide

  148. -OͰࢦఆͨ͠ϑΝΠϧ͕ଘࡏ
    (ϑΝΠϧαΠζ͕1Ҏ্)
    -cΛࢦఆ͍ͯ͠Δ
    ্هͷ৚݅Λຬͨ͢ͱ͖
    206Λॲཧͯ͘͠ΕΔ

    View Slide

  149. ͭ·Γ
    ద౰ͳ1byteͷϑΝΠϧʹ
    -cͰࢦఆ͢Δ͜ͱͰճආՄೳ

    View Slide

  150. ઌ಄ͷ1byteΛऔΓআ͚Ε͹
    alpineඪ४ͷΈͰ࣮ݱՄೳ

    View Slide

  151. tailͰ΋Մೳ͕ͩ஗͍
    dd skip=1 iflag=skip_bytes
    ͕ߴ଎

    View Slide

  152. alpineඪ४ͷΈͰ࣮ݱ

    View Slide

  153. https://github.com/orisano/rget/blob/master/rget.sh

    View Slide