Container Build Talk

Transcript

1. Container Buildͷ࿩ @orisano
2. Agenda • docker buildͷ࿩ • CI ʹ͓͚Δdocker build • docker buildʹඞཁͩͬͨ΋ͷ
3. Agenda • docker buildͷ࿩ • CI ʹ͓͚Δdocker build • docker buildʹඞཁͩͬͨ΋ͷ
4. EPDLFSDMJ EPDLFSE HTTP
5. EPDLFSDMJ EPDLFSE HTTP Mac LinuxKit
6. EPDLFSDMJ EPDLFSE HTTP Mac LinuxKit $ docker build [build context]
7. EPDLFSDMJ EPDLFSE HTTP Mac LinuxKit $ docker build [build context] directoryҎԼΛ.dockerignore Λߟྀͯ͠tarʹ͢Δ
8. EPDLFSDMJ EPDLFSE HTTP Mac LinuxKit $ docker build [build context] directoryҎԼΛ.dockerignore Λߟྀͯ͠tarʹ͢Δ tar
9. Dockerfile
10. Dockerfile͸ ෳ਺ͷεςʔδΛ࣋ͭ εςʔδ ≒ Πϝʔδ
11. Stage FROM image [AS stage] RUN apk add ca-certificates COPY package.json . COPY package-lock.json . ENV NODE_ENV=production RUN npm ci COPY . . RUN npm run build
12. Dockerfile Stage 1 Stage 2 Stage 3
13. Ұ౓ͷbuildͰ࡞ΕΔͷ͸ 1Πϝʔδ͚ͩ
14. جຊతʹ ࠷ޙͷεςʔδ͕ग़ྗ͞ΕΔ
15. Dockerfile Stage 1 Stage 2 Stage 3
16. —targetͰࢦఆ͢Δͱ ग़ྗΠϝʔδΛܾΊΒΕΔ
17. Dockerfile Stage 1 Stage 2 Stage 3 —target
18. Πϝʔδ͸ ϨΠϠʔͷੵΈॏͶͰ ࡞ΒΕΔ
19. ϨΠϠʔ͕࡞ΒΕΔͷ͸ COPY/ADD, RUN ͷλΠϛϯά
20. ϕʔεΠϝʔδ
21. ϕʔεΠϝʔδ $01:
    
    "%%
22. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/
23. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%%
24. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/
25. 36/ tar app/testdata app/.wh.large_file app/a.txt app/b.txt
26. ϨΠϠʔ͸ ࠩ෼ͷϑΝΠϧΛ࣋ͭtar
27. ࡟আ͸.wh.͔Β࢝·Δ whiteoutϑΝΠϧΛ࡞Δ͚ͩ
28. ίϚϯυΛލ͍Ͱ࡟আͯ͠΋ ࢒Γଓ͚Δ
29. ϨΠϠʔ͸ ਌ͷࢀরΛ͚ͩΛ࣋ͭ
30. ୯ҰΠϝʔδͷ৔߹
31. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/
32. ෳ਺Πϝʔδͷ৔߹
33. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/ $01:
    
    "%% 36/ $01:
    
    "%%
34. Build Cache ͷ ϝΧχζϜ
35. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/ $01:
    
    "%% 36/ $01:
    
    "%% ίϐʔͨ͠಺༰Ͱ ίϚϯυ͕มΘΔ
36. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/ $01:
    
    "%% 36/ $01:
    
    "%% ίϐʔͨ͠಺༰Ͱ ίϚϯυ͕มΘΔ ڞ௨ͷ਌Λ࣋ͭಉ͡ίϚϯυͷΠϝʔδͰ ࠷৽ͷ΋ͷΛΩϟογϡͱͯ͠༻͍Δ
37. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/ $01:
    
    "%% 36/ $01:
    
    "%% ίϐʔͨ͠಺༰Ͱ ίϚϯυ͕มΘΔ ڞ௨ͷ਌Λ࣋ͭಉ͡ίϚϯυͷΠϝʔδͰ ࠷৽ͷ΋ͷΛΩϟογϡͱͯ͠༻͍Δ Dockerd
38. άϥϑͱͯ͠ ঢ়ଶΛ͍࣋ͬͯΔͷ͸ dockerd
39. Agenda • docker buildͷ࿩ • CI ʹ͓͚Δdocker build • docker buildʹඞཁͩͬͨ΋ͷ
40. CI্Ͱdocker build cache͕ޮ͔ͳ͍
41. ͳͥͳΒ dockerd͕ͣͬͱੜ͖͍ͯΔ Θ͚Ͱ͸ͳ͍͔Β
42. ڞ௨ͷ਌Λ࣋ͭ ΠϝʔδͳͲ͍ͳ͍
43. ղܾࡦ
44. docker pullͯ͘͠Δ
45. લʹbuildͨ͠ ΠϝʔδΛpull͓͚ͯ͠͹ cache͕ޮ͘?
46. ͍͍͑
47. ֎෦͔Β͖࣋ͬͯͨΠϝʔδ ͸ —cache-from͕ͳ͍ͱର৅֎
48. ͜ΕͰղܾʂ
49. ͦΜͳ࣌୅͕͋Γ·ͨ͠
50. ࠓ͸ େmulti stage build࣌୅
51. աڈͷৗࣝ͸ (෦෼తʹ)௨༻͠ͳ͍
52. εςʔδ͝ͱʹ Πϝʔδ͕࡞ΒΕΔ
53. Dockerfile Stage 1 Stage 2 Stage 3
54. Ұ౓ͷbuildͰ࡞ΕΔͷ͸ 1Πϝʔδ͚ͩ
55. Dockerfile Stage 1 Stage 2 Stage 3 —target
56. cacheͷ࠷খ୯Ґ͕ Πϝʔδ
57. ಛఆͷεςʔδΛ cacheΛޮ͔ͤͯbuild͢Δ ͨΊʹ͸ Ҏલͷεςʔδ͕શͯඞཁ
58. શ෦pull શ෦cache-from
59. docker pull application:build-base-cache || true docker build -t application:build-base-cache --target=build-base --cache- from=application:build-base-cache . docker pull application:base-cache || true docker build -t application:base-cache --target=base --cache-from=application:build-base- cache,application:base-cache . docker pull application:app-build-cache || true docker build -t application:app-build-cache --target=app-build --cache- from=application:build-base-cache,application:base-cache,application:app-build-cache . docker pull application:app-base-cache || true docker build -t application:app-base-cache --target=app-base --cache- from=application:build-base-cache,application:base-cache,application:app-build- cache,application:app-base-cache . docker pull application:application-cache || true docker build -t application:application-cache --target=application --cache- from=application:build-base-cache,application:base-cache,application:app-build- cache,application:app-base-cache,application:application-cache .
60. ߇͑Ίʹ͍ͬͯ ஍ࠈ
61. Agenda • docker buildͷ࿩ • CI ʹ͓͚Δdocker build • docker buildʹඞཁͩͬͨ΋ͷ
62. ඞཁͩͬͨ΋ͷ͸ͳʹ͔
63. docker buildҎ֎ͷ πʔϧΛݟͯΈΔ
64. uber/makisuͷ࿩
65. makisu͸ Uber͕࡞ͬͨbuilder
66. https://eng.uber.com/makisu/
67. 2015೥͘Β͍͔Β DockerҠߦΛ࢝ΊͨUber
68. Apache MesosͱK8S
69. 400αʔϏε͘Β͍
70. docker buildͷprocess ࣗಈԽ, ඪ४Խ
71. ػີ৘ใͷऔѻʹ·ͣࠔͬͨ
72. docker-squashͰղܾʂ
73. ͔͠͠ build͕࣌ؒ2ഒʹ
74. ࢖͍෺ʹͳΒͳ͍ͷͰ dockerΛfork͢Δ͜ͱʹ
75. build࣌ʹ volumeΛmountͰ͖ΔΑ͏ʹ
76. େຬ଍
77. 2017೥ 3000αʔϏε·Ͱ੒௕
78. buildʹ2͔͔࣌ؒΓ 10GBΛ௒͑ΔΠϝʔδ΋
79. storage, ଳҬ, ੜ࢈ੑʹ μϝʔδ
80. εέʔϧ͢Δ ࣍ੈ୅ͷϏϧυʹ͍ͭͯߟ͑Δ
81. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ
82. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ
83. 2017೥ʹ͸ ࣗಈԽ͞Ε, εέʔϥϒϧͰ ޮ཰తͳΠϯϑϥ͕͋ͬͨ
84. Docker build΋ ͦͷ্Ͱಈ͔͔ͨͬͨ͠
85. docker build͸ copy-on-writeͰࠩ෼Λ ͍Ζ͍Ζ΍͍ͬͯΔͷͰ ڧ͍ݖݶ͕ඞཁ
86. ڧ͍ΫϥελͰ ηΩϡϦςΟ໰୊͸ ճආ͍ͨ͠
87. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ
88. LayerCacheͰ લճͷLayerΛ࢖͍ճͤΔͱ build࣌ؒΛ୹͘Ͱ͖Δ
89. Dockerͷcache͸ ϒϥϯνؒ΍ผαʔϏεͩͱ ޮ͔ͳ͍
90. build machineͷׂ౰Ͱ cache hit཰Λ޲্͕ͤͨ͞ ෳࡶ౓্͕͕ͬͯ͠·ͬͨ
91. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ
92. খ͍͞Πϝʔδ͸ ࠷ߴ
93. storage network decompress ʹޮ͘
94. multi stage build͸ ྑ͍ղܾࡦ
95. ͔͠͠ Dockerfile͕ෳࡶʹͳΔ
96. ͦͷ΄͔ʹ΋ layerΛ·͍ͨͰ࡟আ͍ͯ͠Δ ΠϝʔδͳͲ΋͋ͬͨ
97. makisu͸ 3ͭͷ໰୊Λղܾ͢Δ
98. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ
99. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ
100. makisu͸ copy on writeΛ࢖Θͣ In Memory FSͰࠩ෼ΛऔΔ ৄ͘͠͸ޙड़
101. ऄ଍
102. Docker͸layerͷѹॖʹ GoͷgzipΛ࢖͍ͬͯΔ͕ pgzipͷ΄͏͕଎͍ͷͰ makisu͸ͦͬͪ
103. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ
104. Redis(or FS or HTTP)ʹ digestͱRegistryͷ ώϞ෇͕͋Γ Cache͕ղܾ͞ΕΔ
105. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ
106. ಠࣗͷDockerfile parserΛ ࢖ͬͯ ໌ࣔతʹϨΠϠʔΛ࡞Δ ػߏ͕͋Δ
107. Stage FROM image [AS stage] RUN apk add ca-certificates COPY package.json . COPY package-lock.json . ENV NODE_ENV=production RUN npm ci #!COMMIT COPY . . RUN npm run build
108. Stage FROM image [AS stage] RUN apk add ca-certificates COPY package.json . COPY package-lock.json . ENV NODE_ENV=production RUN npm ci #!COMMIT COPY . . RUN npm run build
109. ͜ͷػߏ͕͋Ε͹ػີ৘ใΛ COPYͨ͋͠ͱʹ࡟আͯ͠ COMMITΛ͢Δͱ͍͏ࣄ͕ Dockerfile͚ͩͰ࣮ݱՄೳ
110. kaniko
111. kaniko͸ Google͕࡞ͬͨbuilder
112. ίϯςφ্Ͱಈ͘
113. ͜Ε·Ͱίϯςφ্Ͱͷ buildͬͯͲ͏ͯͨ͠ͷʁ
114. Privileged Container EPDLFSDMJ EPDLFSE HTTP
115. Container EPDLFSDMJ EPDLFSE HTTP Host
116. Container EPDLFSDMJ EPDLFSE HTTP Remote
117. kaniko
118. Container LBOJLP SPPUGT
119. Container LBOJLP SPPUGT 4
     
     ($4 State &$3
     
     ($3 Layer Cache
120. Container LBOJLP SPPUGT 4
     
     ($4 State &$3
     
     ($3 Layer Cache exec
121. Container LBOJLP SPPUGT 4
     
     ($4 State &$3
     
     ($3 Layer Cache exec snapshot
122. Container LBOJLP SPPUGT 4
     
     ($4 State &$3
     
     ($3 Layer Cache exec snapshot
123. Container LBOJLP 4
     
     ($4 State &$3
     
     ($3 SPPUGT &$3
     
     ($3 Layer Cache Image exec snapshot Registry
124. ঢ়ଶΛ ίϯςφ಺෦ʹ࣋ͨͳ͍
125. Image୯ҐͰͷ Pull/CacheͰ͸ͳ͘ Layer୯Ґ
126. BuildͱCacheͷߋ৽͕ ಉ࣌ʹߦΘΕΔ
127. Multi Stage Build Ͱ΋Cache͕ޮ͘
128. Image͸ઈରPush͢Δ
129. ίϯςφΛ࢖ͬͨ Build/Pushʹద͍ͯ͠Δ
130. kanikoʹ͍ͭͯৄ͘͠͸ kaniko ͕ԿΛ͍ͯ͠Δ͔, Կ͕Ͱ͖Δ͔ https://orisano.hatenablog.com/entry/2019/05/20/120032
131. makisuͱkaniko͸ جຊతʹಉ͡࡞Γ
132. kaniko ͷ΄͏͕ Redisͱ͔͍Βͳ͍ͷͰ ࢖͍΍͍͢
133. kaniko͓͢͢ΊͰ͢
134. Dockerfileͷ͜ͱ΍ Docker buildͷ͜ͱͰ ࠔͬͨΒؾܰʹ twitter: @orisano