Better Docker Image+Bonfire Backend #2 #yjbonfire@orisano
View Slide
ྑ͍Docker ImageͱԿ͔
ݟͯΘ͔Γ͍͢खݩͰߴʹbuild͞ΕΔCI্Ͱߴʹbuild͞ΕΔߴʹdeploy͞ΕΔ
ࠓ͍ͨ͜͠ͱ
ͲͷΑ͏ʹ͘͢Δ͔ͲͷΑ͏ʹখ͘͢͞Δ͔
ͲͷΑ͏ʹ͘͢Δ͔• ΠϝʔδΛখ͘͢͞Δ• ίϚϯυͦͷͷΛ͘͢Δ• cacheΛޮ͔ͤΔ• ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ• ඞཁͳ͍εςʔδΛbuild͠ͳ͍
imageΛখ͘͢͞Δͷdocker push͢Δͱ͖ͷdocker pull͢Δͱ͖ͷͷߴԽʹͭͳ͕Δ
Ͳͷ༷ʹখ͘͢͞Δ͔͋ͱͰ
ίϚϯυͦͷͷΛ͘͢Δ
URLʹର͢ΔADDΛΘͳ͍
URLʹର͢ΔADDجຊతʹμϯϩʔυ͢Δ
͍
ΞΫηε͍ͯ͠Δઌͷίϯςϯπ͕ႈͳΒwget + gzip + tarͰे
`RUN wget`ʹ͢Δ͜ͱͰcache͕ޮ͘
ႈͰͳ͍Ϧιʔεʹґଘ͢ΔͷΛۃྗΊΔ
build contextͷసૹྔΛߟ͑Δ
ϞϊϨϙʹ͢Δͱbuild contextେ͖͘ͳΓ͕ͪ
ϞϊϨϙͷ߹.dockerignoreΛࣗಈੜ͠Α͏
github.com/orisano/dignore• ࢦఆ͞ΕͨdirectoryҎ֎ͷdirectoryΛignore• ࢦఆ͞Εͨdirectoryʹdockerignore͕͋Εల։• ͜Ε͚ͩʂ
buildͷલఔͰbuildʹඞཁͳ͍αʔϏεIgnore͠Α͏
buildkitΛ͍ͬͯΔͱࠩసૹͯ͘͠ΕͨΓ͢Δ
COPYΛҙࣝͨ͠directoryߏ
COPYͷҾ͕directoryͷ߹ରͷdirectoryʹதΛશ෦ίϐʔͯ͠͠·͏
ಛఆͷσΟϨΫτϦ͚ͩίϐʔͨ͘͠ͳ͍ͱ͍͏͕͍͠
dockerignoreͰআ֎͢Εྑ͘ͳ͍ʁ
͍͍͑
vendorͳͲΛཧͯ͠build࣌ʹdownload͠ͳ͍߹ʹຊʹਏ͍
2ճҎ্ॏ͍directoryΛCOPYͨ͘͠ͳ͍
ͦ͏ͳΒͳ͍ͨΊͷdirectoryߏʹ͓ͯ͜͠͏
ਓ͕ؒॻ͍ͨͷ͕ೖ͍ͬͯΔdirectory֎෦ʹґଘ͍ͯ͠Δͷࣗಈੜ͕ೖ͍ͬͯΔdirectoryΛ͠Α͏
buildkitͩͱࠩసૹ
͍ͣΕʹͤΑ͓͍ͯͨ͠ํ͕cacheʹ༏͍͠
RUNͷதΛ͘͢Δ
GitHub͔Β࣮ߦϑΝΠϧΛcurl or wgetͰऔಘ͢Δ͕͍
Ͳ͏ʹ͔͘Ͱ͖ͳ͍͔curl -vvvΛ͏ͬͯΈͨ
GitHub ReleaseS3Ͱ͋ΔS3Accept-Ranges: bytesͱ͍͏͜ͱ͕Θ͔ͬͨ
Accept-Ranges: bytesͬͯͳΜͩΖ͏
RFC7233, Range RequestsൣғΛࢦఆͯ͠downloadͰ͖Δ
ͭ·Γฒྻμϯϩʔυ͕Մೳ
github.com/orisano/rget
ڥʹΑΓ·͕͢5min -> 2min
DockerfileͷҎ֎ʹ͘͢ΔΞϓϩʔν͋Δ
cacheΛޮ͔ͤΔ
docker build͢ΔϚγϯ͕ಉҰͷ߹ಉҰͰͳ͍߹
cacheͷΈΛཧղ͢Δ
RUNจࣈྻ͕มΘΒͳ͍ݶΓجຊతʹcache͞ΕΔ
COPY, ADD͢ΔϑΝΠϧͷ༰͕มΘͬͨ߹Ҏ߱ͷRUNͷcache͕ഁغ
ͳͷͰlockfileͳͲΛઌʹίϐʔͯ͠install͚ͩͯ͠cacheͤ͞Δ
CI্Ͱbuild͢Δͱ͖ͳͲcache͕ͳ͍͜ͱ͕ଟ͍
docker save & loadordocker pulldocker build —cache-fromΛ͏
image͕େ͖͍/layer͕ଟ͍߹buildͷ΄͏͕ૣ͍͜ͱ͋Δ
image͕େ͖͍/layer͕ଟ͍߹buildͷ΄͏͕ૣ͍͜ͱ͋Δܭଌܾͯ͠ΊΔ
ґଘͷͳ͍εςʔδΛฒྻͰ࣮ߦ͢Δ
buildkitΛ͍ͬͯͩ͘͞ʂgithub.com/moby/buildkitexport DOCKER_BUILDKIT=1
ඞཁͳ͍εςʔδΛbuild͠ͳ͍
—targetΛ͑Α͍ͷͰʁ
—targetࢦఆͨ͠εςʔδҎલΛશ෦build͢Δ
multi stage buildΛ౿ΈࠐΜ͍ͩํ͍ͯ͠Δͱૺ۰͕ͪ͠
multi stage buildͷ౿ΈࠐΜ͍ͩํʹ͍ͭͯ
https://speakerdeck.com/orisano/multi-stage-builds-patterns-and-practice
͓ͦΒ͘buildkitͰͰ͖Δ͕buildkit͕͑ͳ͍߹ʹ
github.com/orisano/targd• DockerfileͷASTΛऔಘ• ࢦఆ͞ΕͨεςʔδͷґଘεςʔδΛநग़• ґଘεςʔδ͚ͩͷDockerfileΛग़ྗ
github.com/orisano/targd
ͲͷΑ͏ʹখ͘͢͞Δ͔• multi stage buildΛ͏• RUNΛ·ͱΊΔ(?)• ͳͥେ͖͍͔ΛΔ
multi stage buildΛ͏
multi stage buildͰ͋Γ͕ͪͳٙ
࠷ऴతͳΠϝʔδ͕খ͘͞ͳΔ͔ΒͦΕ·Ͱͷεςʔδ࠷దԽ͠ͳͯ͘ྑ͍ʁ
ݸਓతʹNo
moby/issues/34715—cache-formͩͱmulti stage buildͷલஈͷεςʔδͷcache͕ޮ͔ͳ͍
࠷ऴεςʔδ͔͠pushͯ͠ͳ͍͔ΒͨΓલ
CI্Ͱͷbuild—cache-fromΛ͏࠷ऴεςʔδ͚ͩcache͢Δҙຯͳ͍
multi stage build࣌ʹcacheΛޮ͔͍ͤͨ߹લͷεςʔδ໌ࣔతʹpush͢Δ͔͠ͳ͍
͜ΕΛखಈͰΔͱcache-fromࠈʹͳΔ
github.com/orisano/castage• DockerfileͷASTΛऔಘ• εςʔδҰཡΛऔಘ• docker pullΛੜ• docker buildΛcache-from͖Ͱੜ
݁ہpush͢ΔͷͰpush/pullͷίετ͕͔͔Δͯ͢ͷεςʔδΛฏʹখ͖͘͢͞(ݸਓͷݟղͰ͢)
RUNΛ·ͱΊΔ(?)
Ͳ͔ͬʔ;͍͊Δ1ͭͷRUNʹશ෦ॻ͘ͱྑ͍Β͍͠(?)
github.com/orisano/minid• DockerfileͷASTΛऔಘ• ࿈ଓ͢ΔRUN, COPY, ADDΛ࿈݁͢Δ• ݁ՌͷDockerfileΛग़ྗ
github.com/orisano/minid
αΠζ͕খ͘͞ͳͬͨ
ͰͳΜͰʁ
ϨΠϠʔͷΦʔόʔϔου͕ݮΔ͔Βখ͘͞ͳΔʁ
·ͣͲ͏͍͏ܗͰอଘ͞ΕͯΔ͔ΛΔ
moby/image/spec/v1.md
AUFS
http://docs.docker.jp/engine/userguide/storagedriver/aufs-driver.html
আwhiteoutϑΝΠϧͷՃҠಈopaqueϑΝΠϧͷՃʴҠಈઌͷϑΝΠϧࠩ
ҰͰRUN,COPY,ADDΛލ͙ͱimageʹͬͯ͠·͏
ػցతʹͰҰͭͷRUNʹ·ͱΊΔ͜ͱͰ༨ܭͳͷΛݮͰ͖Δ
1ͭͷϨΠϠʔʹ·ͱΊΔ͜ͱຊʹਖ਼͍͠ͷ͔
ϨΠϠʔΛผ͚Δ͜ͱʹΑΓฒྻμϯϩʔυͷԸܙcacheͷ༗ޮ׆༻
ϨΠϠʔΛผ͚Δ͜ͱʹΑΓฒྻμϯϩʔυͷԸܙcacheͷ༗ޮ׆༻ܭଌܾͯ͠ΊΔ
ͳͥେ͖͍͔ΛΔ
͍͔ͳΔνϡʔχϯάͰܭଌͤͣʹ͍͚ͬͯͳ͍
docker history
ͲͷϨΠϠʔ͕େ͖͍͔Θ͔Δ͚Ͳͳͥେ͖͍͔͔ΓͮΒ͍
ҙ֎ʹίϚϯυ͕Ͳ͏͍͏ϑΝΠϧΛ࡞Δͷ͔Βͳ͍
github.com/orisano/dlayer• docker saveͰಘΒΕͨtarΛղੳ• layerͱcmdͷରԠΛͱΔ• layerͷࠩΛϑΝΠϧαΠζ͕େ͖͍ॱʹදࣔ
github.com/orisano/dlayer
࣮ࡍʹdlayerΛͬͯgolangެࣜalpineΠϝʔδͷαΠζΛݮΒͯ͠Έͨ
docker-library/golang/pull/232
golang:1.11͔ΒtoolchainͷαΠζ͕Ͱ͔͘ͳͬͨSSAͰͷ࠷దԽ͕૿͔͑ͨΒʁ
253MB
305MB
github.com/wagoodman/dive
࡞ͬͨπʔϧ• github.com/orisano/rget• github.com/orisano/targd• github.com/orisano/minid• github.com/orisano/dlayer• ྑ͍ͱࢥͬͨΒελʔ͍͚ͯͨͩ͠ΔͱྭΈʹͳΓ·͢
࡞ͬͨπʔϧ• github.com/orisano/dignore• github.com/orisano/castage• ྑ͍ͱࢥͬͨΒελʔ͍͚ͯͨͩ͠ΔͱྭΈʹͳΓ·͢
·ͱΊ• ίϚϯυΛ͘Ͱ͖ͳ͍͔ߟ͑Δ• cacheΛཧղͯ͠༗ޮʹ׆͔͢ (CI or ϩʔΧϧ)• multi stage buildΛ͏ (ॏ͍stageΛcache͢Δ)• layerΛখ͘͢͞ΔͨΊʹੳ͢Δ• buildkitΛ͏!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
rget͜΅Ε
Ͱ͜ΕͰຊʹ͍͍ΜͩΖ͏͔
rgetͷbinaryΛdownload͢ΔͷͰෛ͚ͨؾ͕͢Δ()
shell͚ͩͰ࣮ݱͰ͖ͳ͍ͩΖ͏͔
ϙʔλϒϧੑ͍Βͳ͍ͷͰओઓͷalpineʹߜΔ
alpine(busybox)xargs͕ೖ͍ͬͯΔʂʂʂ
xargsΛ͏͜ͱͰฒྻԽಉ࣌ଓ੍ݶ͕࣮ݱͰ͖Δ
GitHub Release؆୯ʹHEAD͕Ͱ͖ͳ͍ͷͰGETͰbodyΛແࢹͯ͠Content-LengthΛऔಘͰ͖Δ
seqͰchunkͷrangeΛੜͰ͖Δ
wget206 Partial Content͕͏·͘ѻ͑ͳ͍ͷͰμϝ
ํͳ͍ͷͰcurl -RͰdownload
࠷ޙʹcatͯ͠chunkΛ݁߹
shellͰ࣮ݱͰ͖Δʂ
͔͠͠curlalpineඪ४Ͱͳ͍
apk add —no-cache curlෛ͚ͨؾ͕͢Δ()
͔͠͠wget206͕ॲཧͰ͖ͳ͍
ఘΊΒΕͳ͍ͷͰbusyboxͷwgetͷιʔείʔυΛಡΉ
busybox/networking/wget.c
-OͰࢦఆͨ͠ϑΝΠϧ͕ଘࡏ(ϑΝΠϧαΠζ͕1Ҏ্)-cΛࢦఆ͍ͯ͠Δ্هͷ݅Λຬͨ͢ͱ͖206Λॲཧͯ͘͠ΕΔ
ͭ·Γదͳ1byteͷϑΝΠϧʹ-cͰࢦఆ͢Δ͜ͱͰճආՄೳ
ઌ಄ͷ1byteΛऔΓআ͚Εalpineඪ४ͷΈͰ࣮ݱՄೳ
tailͰՄೳ͕͍ͩdd skip=1 iflag=skip_bytes͕ߴ
alpineඪ४ͷΈͰ࣮ݱ
https://github.com/orisano/rget/blob/master/rget.sh
orisano
bun913
amarelo_n24
takashiinui
sat
sbtechnight
kaznishi
makamaka
asazutaiga
datarian
dena_tech
arthur1
renjikari
dotmariusz
brettharned
sugarenia
cassininazir
thoeni
speakerdeck
tmm1
kneath
chriscoyier
samanthasiow
gr2m
nonsquared