Container Build Talk

Transcript

1. Container Buildͷ࿩ @orisano

2. Agenda • docker buildͷ࿩ • CI ʹ͓͚Δdocker build • docker

   buildʹඞཁͩͬͨ΋ͷ

3. Agenda • docker buildͷ࿩ • CI ʹ͓͚Δdocker build • docker

   buildʹඞཁͩͬͨ΋ͷ

4. EPDLFSDMJ EPDLFSE HTTP

5. EPDLFSDMJ EPDLFSE HTTP Mac LinuxKit

6. EPDLFSDMJ EPDLFSE HTTP Mac LinuxKit $ docker build [build context]

7. EPDLFSDMJ EPDLFSE HTTP Mac LinuxKit $ docker build [build context]

   directoryҎԼΛ.dockerignore Λߟྀͯ͠tarʹ͢Δ

8. EPDLFSDMJ EPDLFSE HTTP Mac LinuxKit $ docker build [build context]

   directoryҎԼΛ.dockerignore Λߟྀͯ͠tarʹ͢Δ tar

9. Dockerfile

10. Dockerfile͸ ෳ਺ͷεςʔδΛ࣋ͭ εςʔδ ≒ Πϝʔδ

11. Stage FROM image [AS stage] RUN apk add ca-certificates COPY

    package.json . COPY package-lock.json . ENV NODE_ENV=production RUN npm ci COPY . . RUN npm run build

12. Dockerfile Stage 1 Stage 2 Stage 3

13. Ұ౓ͷbuildͰ࡞ΕΔͷ͸ 1Πϝʔδ͚ͩ

14. جຊతʹ ࠷ޙͷεςʔδ͕ग़ྗ͞ΕΔ

15. Dockerfile Stage 1 Stage 2 Stage 3

16. —targetͰࢦఆ͢Δͱ ग़ྗΠϝʔδΛܾΊΒΕΔ

17. Dockerfile Stage 1 Stage 2 Stage 3 —target

18. Πϝʔδ͸ ϨΠϠʔͷੵΈॏͶͰ ࡞ΒΕΔ

19. ϨΠϠʔ͕࡞ΒΕΔͷ͸ COPY/ADD, RUN ͷλΠϛϯά

20. ϕʔεΠϝʔδ

21. ϕʔεΠϝʔδ $01:
    
    "%%

22. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/

23. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%%

24. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/

25. 36/ tar app/testdata app/.wh.large_file app/a.txt app/b.txt

26. ϨΠϠʔ͸ ࠩ෼ͷϑΝΠϧΛ࣋ͭtar

27. ࡟আ͸.wh.͔Β࢝·Δ whiteoutϑΝΠϧΛ࡞Δ͚ͩ

28. ίϚϯυΛލ͍Ͱ࡟আͯ͠΋ ࢒Γଓ͚Δ

29. ϨΠϠʔ͸ ਌ͷࢀরΛ͚ͩΛ࣋ͭ

30. ୯ҰΠϝʔδͷ৔߹

31. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/

32. ෳ਺Πϝʔδͷ৔߹

33. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/ $01:
    
    "%% 36/ $01:
    
    "%%

34. Build Cache ͷ ϝΧχζϜ

35. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/ $01:
    
    "%% 36/ $01:
    
    "%% ίϐʔͨ͠಺༰Ͱ

    ίϚϯυ͕มΘΔ

36. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/ $01:
    
    "%% 36/ $01:
    
    "%% ίϐʔͨ͠಺༰Ͱ

    ίϚϯυ͕มΘΔ ڞ௨ͷ਌Λ࣋ͭಉ͡ίϚϯυͷΠϝʔδͰ ࠷৽ͷ΋ͷΛΩϟογϡͱͯ͠༻͍Δ

37. ϕʔεΠϝʔδ $01:
    
    "%% ϕʔεΠϝʔδ 36/ $01:
    
    "%% 36/ $01:
    
    "%% 36/ $01:
    
    "%% ίϐʔͨ͠಺༰Ͱ

    ίϚϯυ͕มΘΔ ڞ௨ͷ਌Λ࣋ͭಉ͡ίϚϯυͷΠϝʔδͰ ࠷৽ͷ΋ͷΛΩϟογϡͱͯ͠༻͍Δ Dockerd

38. άϥϑͱͯ͠ ঢ়ଶΛ͍࣋ͬͯΔͷ͸ dockerd

39. Agenda • docker buildͷ࿩ • CI ʹ͓͚Δdocker build • docker

    buildʹඞཁͩͬͨ΋ͷ

40. CI্Ͱdocker build cache͕ޮ͔ͳ͍

41. ͳͥͳΒ dockerd͕ͣͬͱੜ͖͍ͯΔ Θ͚Ͱ͸ͳ͍͔Β

42. ڞ௨ͷ਌Λ࣋ͭ ΠϝʔδͳͲ͍ͳ͍

43. ղܾࡦ

44. docker pullͯ͘͠Δ

45. લʹbuildͨ͠ ΠϝʔδΛpull͓͚ͯ͠͹ cache͕ޮ͘?

46. ͍͍͑

47. ֎෦͔Β͖࣋ͬͯͨΠϝʔδ ͸ —cache-from͕ͳ͍ͱର৅֎

48. ͜ΕͰղܾʂ

49. ͦΜͳ࣌୅͕͋Γ·ͨ͠

50. ࠓ͸ େmulti stage build࣌୅

51. աڈͷৗࣝ͸ (෦෼తʹ)௨༻͠ͳ͍

52. εςʔδ͝ͱʹ Πϝʔδ͕࡞ΒΕΔ

53. Dockerfile Stage 1 Stage 2 Stage 3

54. Ұ౓ͷbuildͰ࡞ΕΔͷ͸ 1Πϝʔδ͚ͩ

55. Dockerfile Stage 1 Stage 2 Stage 3 —target

56. cacheͷ࠷খ୯Ґ͕ Πϝʔδ

57. ಛఆͷεςʔδΛ cacheΛޮ͔ͤͯbuild͢Δ ͨΊʹ͸ Ҏલͷεςʔδ͕શͯඞཁ

58. શ෦pull શ෦cache-from

59. docker pull application:build-base-cache || true docker build -t application:build-base-cache --target=build-base

    --cache- from=application:build-base-cache . docker pull application:base-cache || true docker build -t application:base-cache --target=base --cache-from=application:build-base- cache,application:base-cache . docker pull application:app-build-cache || true docker build -t application:app-build-cache --target=app-build --cache- from=application:build-base-cache,application:base-cache,application:app-build-cache . docker pull application:app-base-cache || true docker build -t application:app-base-cache --target=app-base --cache- from=application:build-base-cache,application:base-cache,application:app-build- cache,application:app-base-cache . docker pull application:application-cache || true docker build -t application:application-cache --target=application --cache- from=application:build-base-cache,application:base-cache,application:app-build- cache,application:app-base-cache,application:application-cache .

60. ߇͑Ίʹ͍ͬͯ ஍ࠈ

61. Agenda • docker buildͷ࿩ • CI ʹ͓͚Δdocker build • docker

    buildʹඞཁͩͬͨ΋ͷ

62. ඞཁͩͬͨ΋ͷ͸ͳʹ͔

63. docker buildҎ֎ͷ πʔϧΛݟͯΈΔ

64. uber/makisuͷ࿩

65. makisu͸ Uber͕࡞ͬͨbuilder

66. https://eng.uber.com/makisu/

67. 2015೥͘Β͍͔Β DockerҠߦΛ࢝ΊͨUber

68. Apache MesosͱK8S

69. 400αʔϏε͘Β͍

70. docker buildͷprocess ࣗಈԽ, ඪ४Խ

71. ػີ৘ใͷऔѻʹ·ͣࠔͬͨ

72. docker-squashͰղܾʂ

73. ͔͠͠ build͕࣌ؒ2ഒʹ

74. ࢖͍෺ʹͳΒͳ͍ͷͰ dockerΛfork͢Δ͜ͱʹ

75. build࣌ʹ volumeΛmountͰ͖ΔΑ͏ʹ

76. େຬ଍

77. 2017೥ 3000αʔϏε·Ͱ੒௕

78. buildʹ2͔͔࣌ؒΓ 10GBΛ௒͑ΔΠϝʔδ΋

79. storage, ଳҬ, ੜ࢈ੑʹ μϝʔδ

80. εέʔϧ͢Δ ࣍ੈ୅ͷϏϧυʹ͍ͭͯߟ͑Δ

81. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ

82. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ

83. 2017೥ʹ͸ ࣗಈԽ͞Ε, εέʔϥϒϧͰ ޮ཰తͳΠϯϑϥ͕͋ͬͨ

84. Docker build΋ ͦͷ্Ͱಈ͔͔ͨͬͨ͠

85. docker build͸ copy-on-writeͰࠩ෼Λ ͍Ζ͍Ζ΍͍ͬͯΔͷͰ ڧ͍ݖݶ͕ඞཁ

86. ڧ͍ΫϥελͰ ηΩϡϦςΟ໰୊͸ ճආ͍ͨ͠

87. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ

88. LayerCacheͰ લճͷLayerΛ࢖͍ճͤΔͱ build࣌ؒΛ୹͘Ͱ͖Δ

89. Dockerͷcache͸ ϒϥϯνؒ΍ผαʔϏεͩͱ ޮ͔ͳ͍

90. build machineͷׂ౰Ͱ cache hit཰Λ޲্͕ͤͨ͞ ෳࡶ౓্͕͕ͬͯ͠·ͬͨ

91. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ

92. খ͍͞Πϝʔδ͸ ࠷ߴ

93. storage network decompress ʹޮ͘

94. multi stage build͸ ྑ͍ղܾࡦ

95. ͔͠͠ Dockerfile͕ෳࡶʹͳΔ

96. ͦͷ΄͔ʹ΋ layerΛ·͍ͨͰ࡟আ͍ͯ͠Δ ΠϝʔδͳͲ΋͋ͬͨ

97. makisu͸ 3ͭͷ໰୊Λղܾ͢Δ

98. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ

99. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ

100. makisu͸ copy on writeΛ࢖Θͣ In Memory FSͰࠩ෼ΛऔΔ ৄ͘͠͸ޙड़

101. ऄ଍

102. Docker͸layerͷѹॖʹ GoͷgzipΛ࢖͍ͬͯΔ͕ pgzipͷ΄͏͕଎͍ͷͰ makisu͸ͦͬͪ

103. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ

104. Redis(or FS or HTTP)ʹ digestͱRegistryͷ ώϞ෇͕͋Γ Cache͕ղܾ͞ΕΔ

105. ϙʔλϒϧ ෼ࢄΩϟογϡ αΠζ࠷దԽ

106. ಠࣗͷDockerfile parserΛ ࢖ͬͯ ໌ࣔతʹϨΠϠʔΛ࡞Δ ػߏ͕͋Δ

107. Stage FROM image [AS stage] RUN apk add ca-certificates COPY

     package.json . COPY package-lock.json . ENV NODE_ENV=production RUN npm ci #!COMMIT COPY . . RUN npm run build

108. Stage FROM image [AS stage] RUN apk add ca-certificates COPY

     package.json . COPY package-lock.json . ENV NODE_ENV=production RUN npm ci #!COMMIT COPY . . RUN npm run build

109. ͜ͷػߏ͕͋Ε͹ػີ৘ใΛ COPYͨ͋͠ͱʹ࡟আͯ͠ COMMITΛ͢Δͱ͍͏ࣄ͕ Dockerfile͚ͩͰ࣮ݱՄೳ

110. kaniko

111. kaniko͸ Google͕࡞ͬͨbuilder

112. ίϯςφ্Ͱಈ͘

113. ͜Ε·Ͱίϯςφ্Ͱͷ buildͬͯͲ͏ͯͨ͠ͷʁ

114. Privileged Container EPDLFSDMJ EPDLFSE HTTP

115. Container EPDLFSDMJ EPDLFSE HTTP Host

116. Container EPDLFSDMJ EPDLFSE HTTP Remote

117. kaniko

118. Container LBOJLP SPPUGT

119. Container LBOJLP SPPUGT 4
     
     ($4 State &$3
     
     ($3 Layer Cache

120. Container LBOJLP SPPUGT 4
     
     ($4 State &$3
     
     ($3 Layer Cache exec

121. Container LBOJLP SPPUGT 4
     
     ($4 State &$3
     
     ($3 Layer Cache exec snapshot

122. Container LBOJLP SPPUGT 4
     
     ($4 State &$3
     
     ($3 Layer Cache exec snapshot

123. Container LBOJLP 4
     
     ($4 State &$3
     
     ($3 SPPUGT &$3
     
     ($3 Layer Cache Image

     exec snapshot Registry

124. ঢ়ଶΛ ίϯςφ಺෦ʹ࣋ͨͳ͍

125. Image୯ҐͰͷ Pull/CacheͰ͸ͳ͘ Layer୯Ґ

126. BuildͱCacheͷߋ৽͕ ಉ࣌ʹߦΘΕΔ

127. Multi Stage Build Ͱ΋Cache͕ޮ͘

128. Image͸ઈରPush͢Δ

129. ίϯςφΛ࢖ͬͨ Build/Pushʹద͍ͯ͠Δ

130. kanikoʹ͍ͭͯৄ͘͠͸ kaniko ͕ԿΛ͍ͯ͠Δ͔, Կ͕Ͱ͖Δ͔ https://orisano.hatenablog.com/entry/2019/05/20/120032

131. makisuͱkaniko͸ جຊతʹಉ͡࡞Γ

132. kaniko ͷ΄͏͕ Redisͱ͔͍Βͳ͍ͷͰ ࢖͍΍͍͢

133. kaniko͓͢͢ΊͰ͢

134. Dockerfileͷ͜ͱ΍ Docker buildͷ͜ͱͰ ࠔͬͨΒؾܰʹ twitter: @orisano