Better Docker Image

2018/10/12 フューチャーアーキテクトとのLTバトルでの、與那城（@orisano）の講演資料になります

Transcript

1. Better Docker Image Container Build Meetup #1 #container_build @orisano
2. ྑ͍Docker Imageͱ͸Կ͔
3. αʔϏεʹؔΘΔਓΛ ޾ͤʹ͢ΔΠϝʔδ
4. ݟͯΘ͔Γ΍͍͢ खݩͰߴ଎ʹbuild͞ΕΔ CI্Ͱߴ଎ʹbuild͞ΕΔ ߴ଎ʹdeploy͞ΕΔ
5. ࠓ೔࿩͍ͨ͜͠ͱ
6. ͲͷΑ͏ʹ଎͘͢Δ͔ ͲͷΑ͏ʹখ͘͢͞Δ͔
7. ͲͷΑ͏ʹ଎͘͢Δ͔ ͲͷΑ͏ʹখ͘͢͞Δ͔
8. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
9. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
10. imageΛখ͘͢͞Δͷ͸ docker push͢Δͱ͖ͷ଎౓ docker pull͢Δͱ͖ͷ଎౓ ͷߴ଎Խʹͭͳ͕Δ
11. Ͳͷ༷ʹখ͘͢͞Δ͔͸ ͋ͱͰ
12. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
13. ίϚϯυͦͷ΋ͷΛ଎͘͢Δ
14. URLʹର͢ΔADDΛ࢖Θͳ͍ RUNͷத਎Λ଎͘͢Δ
15. URLʹର͢ΔADDΛ࢖Θͳ͍ RUNͷத਎Λ଎͘͢Δ
16. URLʹର͢ΔADD͸ جຊతʹμ΢ϯϩʔυ͢Δ
17. ஗͍
18. ΞΫηε͍ͯ͠Δઌͷ ίϯςϯπ͕ႈ౳ͳΒ wget + gzip + tarͰे෼
19. `RUN wget`ʹ͢Δ͜ͱͰ cache͕ޮ͘
20. ႈ౳Ͱͳ͍Ϧιʔεʹ ґଘ͢ΔͷΛۃྗ΍ΊΔ
21. URLʹର͢ΔADDΛ࢖Θͳ͍ RUNͷத਎Λ଎͘͢Δ
22. GitHub͔Β࣮ߦϑΝΠϧΛ curl or wgetͰऔಘ͢Δ ͕஗͍
23. Ͳ͏ʹ͔଎͘Ͱ͖ͳ͍͔ curl -vvvΛ͏ͬͯΈͨ
24. GitHub Release͸S3Ͱ͋Δ S3͸Accept-Ranges: bytes ͱ͍͏͜ͱ͕Θ͔ͬͨ
25. Accept-Ranges: bytes ͬͯͳΜͩΖ͏
26. RFC7233, Range Requests ൣғΛࢦఆͯ͠ downloadͰ͖Δ
27. ͭ·Γ ฒྻμ΢ϯϩʔυ͕Մೳ
28. github.com/orisano/rget
30. ؀ڥʹΑΓ·͕͢ 5min -> 2min
31. Dockerfileͷ޻෉Ҏ֎ʹ΋ ଎͘͢ΔΞϓϩʔν͸͋Δ
32. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
33. cacheΛޮ͔ͤΔ
34. docker build͢ΔϚγϯ͕ ಉҰͷ৔߹ ಉҰͰͳ͍৔߹
35. docker build͢ΔϚγϯ͕ ಉҰͷ৔߹ ಉҰͰͳ͍৔߹
36. cacheͷ࢓૊ΈΛཧղ͢Δ
37. RUN͸ จࣈྻ͕มΘΒͳ͍ݶΓ جຊతʹcache͞ΕΔ
38. COPY, ADD͢ΔϑΝΠϧͷ ಺༰͕มΘͬͨ৔߹ Ҏ߱ͷRUNͷcache͕ഁغ
39. ͳͷͰ lockfileͳͲΛઌʹίϐʔͯ͠ install͚ͩͯ͠cacheͤ͞Δ
42. docker build͢ΔϚγϯ͕ ಉҰͷ৔߹ ಉҰͰͳ͍৔߹
43. CI্Ͱbuild͢Δͱ͖ͳͲ cache͕ͳ͍͜ͱ͕ଟ͍
44. docker save & load or docker pull docker build —cache-from Λ࢖͏
45. image͕େ͖͍/layer͕ଟ͍ ৔߹ buildͷ΄͏͕ૣ͍͜ͱ΋͋Δ
46. image͕େ͖͍/layer͕ଟ͍ ৔߹ buildͷ΄͏͕ૣ͍͜ͱ΋͋Δ ܭଌܾͯ͠ΊΔ
47. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
48. ґଘͷͳ͍εςʔδΛ ฒྻͰ࣮ߦ͢Δ
49. buildkitΛ࢖͍ͬͯͩ͘͞ʂ github.com/moby/buildkit export DOCKER_BUILDKIT=1
50. ͲͷΑ͏ʹ଎͘͢Δ͔ • ΠϝʔδΛখ͘͢͞Δ • ίϚϯυͦͷ΋ͷΛ଎͘͢Δ • cacheΛޮ͔ͤΔ • ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ • ඞཁͳ͍εςʔδΛbuild͠ͳ͍
51. ඞཁͳ͍εςʔδΛ build͠ͳ͍
52. —targetΛ࢖͑͹Α͍ͷͰ͸ʁ
53. —target͸ ࢦఆͨ͠εςʔδҎલΛ શ෦build͢Δ
54. multi stage buildΛ ౿ΈࠐΜͩ࢖͍ํ͍ͯ͠Δͱ ૺ۰͕ͪ͠
55. ͓ͦΒ͘buildkitͰͰ͖Δ͕ buildkit͕࢖͑ͳ͍৔߹ʹ
56. github.com/orisano/targd • DockerfileͷASTΛऔಘ • ࢦఆ͞ΕͨεςʔδͷґଘεςʔδΛநग़ • ґଘεςʔδ͚ͩͷDockerfileΛग़ྗ
57. github.com/orisano/targd
58. ͲͷΑ͏ʹ଎͘͢Δ͔ ͲͷΑ͏ʹখ͘͢͞Δ͔
59. ͲͷΑ͏ʹখ͘͢͞Δ͔ • multi stage buildΛ࢖͏ • RUNΛ·ͱΊΔ(?) • ͳͥେ͖͍͔Λ஌Δ
60. ͲͷΑ͏ʹখ͘͢͞Δ͔ • multi stage buildΛ࢖͏ • RUNΛ·ͱΊΔ(?) • ͳͥେ͖͍͔Λ஌Δ
61. multi stage buildΛ࢖͏
62. multi stage buildͰ ͋Γ͕ͪͳٙ໰
63. ࠷ऴతͳΠϝʔδ͕ খ͘͞ͳΔ͔Β ͦΕ·Ͱͷεςʔδ͸ ࠷దԽ͠ͳͯ͘ྑ͍ʁ
65. ݸਓతʹ͸No
66. moby/issues/34715 —cache-formͩͱ multi stage buildͷ લஈͷεςʔδͷcache͕ ޮ͔ͳ͍
67. ࠷ऴεςʔδ͔͠ pushͯ͠ͳ͍͔Β౰ͨΓલ
68. CI্Ͱͷbuild͸ —cache-fromΛ࢖͏ ࠷ऴεςʔδ͚ͩcache͢Δ ҙຯ͸ͳ͍
69. multi stage build࣌ʹ cacheΛޮ͔͍ͤͨ৔߹͸ લͷεςʔδ΋ ໌ࣔతʹpush͢Δ͔͠ͳ͍
70. ݁ہpush͢ΔͷͰ push/pullͷίετ͕͔͔Δ ͢΂ͯͷεςʔδΛฏ౳ʹ খ͘͢͞΂͖ (ݸਓͷݟղͰ͢)
71. ͲͷΑ͏ʹখ͘͢͞Δ͔ • multi stage buildΛ࢖͏ • RUNΛ·ͱΊΔ(?) • ͳͥେ͖͍͔Λ஌Δ
72. RUNΛ·ͱΊΔ(?)
73. Ͳ͔ͬʔ;͍͊Δ͸ 1ͭͷRUNʹ શ෦ॻ͘ͱྑ͍Β͍͠(?)
74. github.com/orisano/minid • DockerfileͷASTΛऔಘ • ࿈ଓ͢ΔRUN, COPY, ADDΛ࿈݁͢Δ • ݁ՌͷDockerfileΛग़ྗ
75. github.com/orisano/minid
76. αΠζ͕খ͘͞ͳͬͨ
77. Ͱ΋ͳΜͰʁ
78. ϨΠϠʔͷΦʔόʔϔου͕ ݮΔ͔Βখ͘͞ͳΔʁ
79. ͍͍͑
80. ·ͣ Ͳ͏͍͏ܗͰอଘ͞ΕͯΔ͔ Λ஌Δ
81. moby/image/spec/v1.md
82. AUFS
83. http://docs.docker.jp/engine/userguide/storagedriver/aufs-driver.html
84. ࡟আ͸ whiteoutϑΝΠϧͷ௥Ճ Ҡಈ͸ opaqueϑΝΠϧͷ௥Ճ ʴ ҠಈઌͷϑΝΠϧࠩ෼
85. Ұ౓Ͱ΋ RUN,COPY,ADDΛލ͙ͱ imageʹ͸࢒ͬͯ͠·͏
86. ػցతʹͰ΋ ҰͭͷRUNʹ·ͱΊΔ͜ͱͰ ༨ܭͳ΋ͷΛ࡟ݮͰ͖Δ
87. 1ͭͷϨΠϠʔʹ ·ͱΊΔ͜ͱ͸ ຊ౰ʹਖ਼͍͠ͷ͔
88. ϨΠϠʔΛผ͚Δ͜ͱʹΑΓ ฒྻμ΢ϯϩʔυͷԸܙ cacheͷ༗ޮ׆༻ Մಡੑͷ޲্
89. ϨΠϠʔΛผ͚Δ͜ͱʹΑΓ ฒྻμ΢ϯϩʔυͷԸܙ cacheͷ༗ޮ׆༻ Մಡੑͷ޲্ ܭଌܾͯ͠ΊΔ
90. ͱ͸͍͑ ෳ਺ϨΠϠʔʹ ލ͕Βͳ͍ఔ౓ʹ ෼ׂ͢Δͷ͕ྑͦ͞͏
91. ͲͷΑ͏ʹখ͘͢͞Δ͔ • multi stage buildΛ࢖͏ • RUNΛ·ͱΊΔ(?) • ͳͥେ͖͍͔Λ஌Δ
92. ͳͥେ͖͍͔Λ஌Δ
93. ͍͔ͳΔνϡʔχϯάͰ΋ ܭଌͤͣʹ΍ͬͯ͸͍͚ͳ͍
94. docker history
95. docker history
96. ͲͷϨΠϠʔ͕ େ͖͍͔͸Θ͔Δ͚Ͳ ͳͥେ͖͍͔͸෼͔ΓͮΒ͍
97. ҙ֎ʹίϚϯυ͕ Ͳ͏͍͏ϑΝΠϧΛ࡞Δͷ͔ ஌Βͳ͍
98. github.com/orisano/dlayer • docker saveͰಘΒΕͨtarΛղੳ • layerͱcmdͷରԠΛͱΔ • layerͷࠩ෼ΛϑΝΠϧαΠζ͕େ͖͍ॱʹද ࣔ
99. github.com/orisano/dlayer
100. ࣮ࡍʹdlayerΛ࢖ͬͯ golangެࣜalpineΠϝʔδͷ αΠζΛݮΒͯ͠Έͨ
101. docker-library/golang/pull/232
103. golang:1.11͔Βtoolchainͷ αΠζ͕Ͱ͔͘ͳͬͨ SSAͰͷ࠷దԽ͕૿͔͑ͨΒʁ
104. 1.10: 253MB
105. 1.11: 305MB
106. ࡞ͬͨπʔϧ • github.com/orisano/rget • github.com/orisano/targd • github.com/orisano/minid • github.com/orisano/dlayer • ྑ͍ͱࢥͬͨΒελʔ͍͚ͯͨͩ͠ΔͱྭΈʹͳΓ·͢
107. ·ͱΊ • ίϚϯυΛ଎͘Ͱ͖ͳ͍͔ߟ͑Δ • cacheΛཧղͯ͠༗ޮʹ׆͔͢ (CI or ϩʔΧϧ) • multi stage buildΛ࢖͏ (ॏ͍stageΛcache͢Δ) • layerΛখ͘͢͞ΔͨΊʹ෼ੳ͢Δ • buildkitΛ࢖͏!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
108. rget͜΅Ε࿩
109. github.com/orisano/rget
110. Ͱ΋ ͜ΕͰຊ౰ʹ͍͍ΜͩΖ͏͔
111. rgetͷbinaryΛdownload ͢ΔͷͰ͸ෛ͚ͨؾ͕͢Δ()
112. shell͚ͩͰ ࣮ݱͰ͖ͳ͍ͩΖ͏͔
113. ϙʔλϒϧੑ͸͍Βͳ͍ͷͰ ओઓ৔ͷalpineʹߜΔ
114. alpine(busybox)͸ xargs͕ೖ͍ͬͯΔʂʂʂ
115. xargsΛ࢖͏͜ͱͰ ฒྻԽ ಉ࣌઀ଓ਺੍ݶ ͕࣮ݱͰ͖Δ
116. GitHub Release͸ ؆୯ʹHEAD͕Ͱ͖ͳ͍ͷͰ GETͰbodyΛແࢹͯ͠ Content-LengthΛऔಘͰ͖Δ
117. seqͰ chunkͷrangeΛੜ੒Ͱ͖Δ
118. wget͸206 Partial Content ͕͏·͘ѻ͑ͳ͍ͷͰμϝ
119. ࢓ํͳ͍ͷͰ curl -RͰdownload
120. ࠷ޙʹcatͯ͠chunkΛ݁߹
121. shellͰ΋࣮ݱͰ͖Δʂ
122. ͔͠͠ curl͸alpineඪ४Ͱ͸ͳ͍
123. apk add —no-cache curl ͸ෛ͚ͨؾ͕͢Δ()
124. ͔͠͠ wget͸206͕ॲཧͰ͖ͳ͍
125. ఘΊΒΕͳ͍ͷͰ busyboxͷwgetͷ ιʔείʔυΛಡΉ
126. busybox/networking/wget.c
127. busybox/networking/wget.c
128. -OͰࢦఆͨ͠ϑΝΠϧ͕ଘࡏ (ϑΝΠϧαΠζ͕1Ҏ্) -cΛࢦఆ͍ͯ͠Δ ্هͷ৚݅Λຬͨ͢ͱ͖ 206Λॲཧͯ͘͠ΕΔ
129. ͭ·Γ ద౰ͳ1byteͷϑΝΠϧʹ -cͰࢦఆ͢Δ͜ͱͰճආՄೳ
130. ઌ಄ͷ1byteΛऔΓআ͚Ε͹ alpineඪ४ͷΈͰ࣮ݱՄೳ
131. tailͰ΋Մೳ͕ͩ஗͍ dd skip=1 iflag=skip_bytes ͕ߴ଎
132. alpineඪ४ͷΈͰ࣮ݱ
133. https://github.com/orisano/rget/blob/master/rget.sh