T3DD23: The Anatomy Of A Docker Image

Over the past few years, container technologies (like Docker) have become essential to many a developer's workflow -- be it for local development, testing, CI or production rollouts. In this talk, I will shed light on one of the underlying principles of Docker: the OCI image and distribution specifications. These describe the file format of a Docker image and the API of a Docker registry. Understanding these will help you understand what's actually going on when you run commands like "docker build" or "docker pull". I will also present a case study of a project in which we used these specifications to build a container image solution that implemented features that would have been impossible to implement using the regular tools.

Transcript

1. Lucas Hoang https://unsplash.com/photos/RJPEfLuv6Ao ANATOMY THE OF A DOCKER IMAGE TYPO3 Developer Days, August 4th, 2023 MARTIN HELMICH @mittwald
2. MARTIN HELMICH Head of Architecture & Chief Tech Evangelist Lecturer, Software Engineering & Cloud Computing Sci-Fi-Nerd, Metalhead, Amateur Woodworker
3. CONTAINERS ARE EVERYWHERE Niklas9416, https://pixabay.com/de/photos/hamburg-hafen-schiff-container-6849995/
4. PROTO TYPING DEV TEST DEPLOY OPS
5. MY PROBLEM kirill_makes_pics https://pixabay.com/photos/despaired-businessman-business-2261021/
6. As a web project owner, I can freely configure system components in my environment, so that I can operate a wide range of different applications with different system requirements.
7. > mw project create -q --update-context --wait > ID=$(mw app install typo3 --version=12.4.4 -q --wait) > mw app dependency update $ID \ --set 'php=^8.2' \ --set 'graphicsmagick=*' \ --set 'nodejs=^20' TASK: BUILD A CONTAINER IMAGE FROM THIS USER-PROVIDED SPECIFICATION UNRELATED: https://github.com/mittwald/cli
8. FROM ubuntu:22.04 RUN apt-get update -y RUN apt-get install –y php RUN apt-get install –y graphicsmagick RUN apt-get install –y nodejs INTRODUCING THE CONTAINERFILE
9. FROM ubuntu:22.04 RUN apt-get update -y {{- range $software, $version := .UserRequirements }} RUN apt-get install -y {{ $software }}={{ $version }} {{- end }} IS THIS A GOOD IDEA? (spoiler alert: it is, indeed, not) (actually, WAY more complicated)
10. php=^8.2 graphicsmagick=* nodejs=^20 FROM ubuntu 22.04 {{- range $sw, $v := .User apt-get install -y {{ $sw {{- end }} USER ENVIRONMENT SPECIFICATION TEMPLATED CONTAINERFILE (actually, WAY more complicated) CONTAINER IMAGE DOCKER BUILD HARBOR DOCKER PUSH DOCKER PULL CONTAINER REGISTRY USER PROFIT 💰
11. > mw project create -q --update-context --wait > ID=$(mw app install typo3 --version=12.4.4 -q --wait) > mw app dependency update $ID \ --set 'php=^8.2' \ --set 'graphicsmagick=*' \ --set 'nodejs=^20‘ ⏳ updating environment… ✅ environment updated after 27m51s UNRELATED: https://github.com/mittwald/cli 🫠
12. kirill_makes_pics https://pixabay.com/photos/despaired-businessman-business-2261021/
13. COMMON CONTAINER BUILD OPTIMIZATION TARGETS 📐 Image size ⏱ Build Time 🏎 Download time yellowcat https://pixabay.com/photos/sport-clock-jogging-time-fitness-3874743/
14. MEH. WHO CARES? 😐
15. LIFECYCLE OF A CONTAINER PULL / RUN PUSH BUILD
16. CONTAINER RUNTIME PULL / RUN BUILD ENVIRONMENT CONTAINER REGISTRY PUSH BUILD IMAGE SPEC https://github.com/opencontainers/image-spec DISTRIBUTION SPEC https://github.com/opencontainers/distribution-spec RUNTIME SPEC https://github.com/opencontainers/runtime-spec
18. file1 file2 file3 file2 file4 FILESYSTEM LAYERS UNION MOUNT (OVERLAYFS or AUFS) file1 file2 file3 file4
19. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y composer php-fpm COPY composer.json src /var/www/html/ RUN composer install --no-dev # to be continued… CONTAINERFILE
20. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y composer php-fpm COPY composer.json src /var/www/html/ RUN composer install --no-dev # to be continued… CACHED UNCACHED A change invalidates all cached layers after this one PRO TIP Pull your base image before building to make sure it’s current #security PRO TIP Most of the time, you don’t need dev dependencies in your container image.
21. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y composer php-fpm COPY composer.json src /var/www/html/ RUN composer install --no-dev # to be continued… This file changes (presumably) relatively frequently This file changes (presumably) relatively infrequently This step needs only to run when the composer.json changed
22. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y composer php-fpm COPY composer.json /var/www/html/ RUN composer install --no-dev COPY src /var/www/html/ # to be continued… CACHED UNCACHED Build instruction order determines cache utilization!
23. FROM ubuntu:22.04 AS builder RUN apt-get update –y RUN apt-get install –y composer COPY composer.json app/ RUN composer install --no-dev FROM ubuntu:22.04 RUN apt-get update –y && apt-get install –y php-fpm COPY --from=builder /var/www/html/vendor ↵ /var/www/html/ COPY src /var/www/html/ # to be continued… CACHED UNCACHED Use multistage builds to optimize even further CACHED
24. composer.json auth.json vendor/ auth.json COPY composer.json auth.json /app RUN composer install --no-dev RUN rm auth.json composer.json auth.json vendor/ DON’T DO THIS! Your credentials are still there!
25. composer.json auth.json vendor/ COPY composer.json auth.json /app RUN composer install --no-dev vendor/ DO THIS, instead And keep your credentials safe! FROM <base> AS builder FROM <base> vendor/ COPY --from=base vendor vendor
26. REGULAR DOCKER USERS ME https://knowyourmeme.com/memes/two-guys-on-a-bus
27. CONFIGURATION (JSON) sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744 FS LAYERS sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7 sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4 sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9 sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216 sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8 sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476 sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736 sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8 sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9 (USUAL FILE FORMATS: TAR or TAR.GZ) CONTENTS OF A DOCKER IMAGE MANIFEST
28. INDEX (JSON) amd64 armv7 arm64 MANIFEST CONFIGURATION (JSON) sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744 FS LAYERS sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7 sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4 sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9 sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216 sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8 sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476 sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736 sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8 sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9 (USUAL FILE FORMATS: TAR or TAR.GZ)
29. amd64 armv7 arm64 GET /v2/{image}/manifests/{tag} Accept: application/vnd.oci.image.index.v1+json { "mediaType": "application/vnd.oci.image.index.v1+json", "schemaVersion": 2, "manifests": [ { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:f10dac435406078cfc491daa0cd99598b10909ae259a4e13d…", "size": 3145, "platform": { "architecture": "arm", "os": "linux", "variant": "v7" } }, […] ] } INDEX (JSON)
30. GET /v2/{image}/manifests/{digest} Accept: application/vnd.oci.image.manifest.v1+json { "mediaType": "application/vnd.oci.image.manifest.v1+json", "schemaVersion": 2, "config": { "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:966da77cd61840d8d91d34b9cbdae2104683f93be5550f9819b…", "size": 14426 }, "layers": [ { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:faef57eae888cbe4a5613eca6741b5e48d768b83f6088858a…", "size": 29124829 }, […] ] } MANIFEST ION 63d1504addbe7eb6b82e7798fb744 S 6b87074e55fea6bd1ecae066693c7 13d066d60543ac1b176802dd8638d 6e956b4c0bd9f0c29aaf1b9c5b49b 9716aa4f8df2d148dc1f947b1abe4 995ba25193b0f43ba8bbc4abd99f9 0daf28b405740eb498f9c07837f4f 007953eb385eb7c8425461fb2a54f a3ede53659006c66744b472a95216 367ede931b8e2a5272c562cdcfee8 7e60cb9b50efb1ead4d05d35f079f 343953d068478845c22870c596476 f5e07dc2fd02f0c1b5931ba2b230d 960290328e8829987721c61143736 f9b1e9aee0d51a5320cd5ba6d60a8 f578a8647c7811807d0a2723916b9 R or TAR.GZ)
31. “IT’S JUST A REST API, ISN’T IT?” r.HandleFunc( "/v2/{repo}/{image}/manifests/{tag}", func(w http.ResponseWriter, r *http.Response) { repo, image, tag := manifestRequestVars(r) if manifest, ok := GetManifest(repo, image, tag); ok { json.NewEncoder(w).Write(manifest) } }, ) r.HandleFunc( "/v2/{repo}/{image}/blobs/{digest}", func(w http.ResponseWriter, r *http.Response) { repo, image, digest := blobRequestVars(r) file, err := OpenBlob(repo, image, digest) if err != nil { // handle error } io.Copy(file, r) }, )
32. php=^8.2 graphicsmagick=* nodejs=^20 USER ENVIRONMENT SPECIFICATION PRE-PACKAGED, INDEPENDENT (!) LAYERS (VIRTUAL) CONTAINER IMAGE GENERATE IMAGE MANIFEST CUSTOM REGISTRY SERVE VIA CUSTOM API DOCKER PULL CONTAINER REGISTRY PROFIT 💰 USER BUILT AHEAD-OF-TIME
33. > mw project create -q --update-context --wait > ID=$(mw app install typo3 --version=12.4.4 -q --wait) > mw app dependency update $ID \ --set 'php=^8.2' \ --set 'graphicsmagick=*' \ --set 'nodejs=^20‘ ⏳ updating environment… ✅ environment updated after 180ms UNRELATED: https://github.com/mittwald/cli 🚀 (APPROXIMATELY)
34. ThMilherou https://pixabay.com/photos/workshop-carpenter-crafts-artisan-4863393/ TOOL RECOMMENDATIONS
35. DIVE (https://github.com/wagoodman/dive) Explore individual layers in a given image
36. SKOPEO (https://github.com/containers/skopeo) Inspect (and manipulate) images in remote registries
37. SLIM TOOLKIT (https://github.com/slimtoolkit/slim) Optimize images for size, using black magic (requires caution!)
38. <insert appropriate last words here>
39. END THE Three-shots https://pixabay.com/photos/ocean-blue-whale-sea-wave-whale-2051760/
40. BONUS SLIDES
41. FROM big-ass-image:v1.2.3 AS source FROM scratch COPY --from=source / / # NOTE: Make sure to set environment variables CMD … ENTRYPOINT … SQUASHING IMAGES
42. SQUASHING IMAGES ORIGINAL IMAGE: 1Gi SQUASHED IMAGE: 400Mi
43. SQUASHING IMAGES ORIGINAL IMAGE: 1Gi DOWNLOAD ON LAYER CHANGE: 10Mi SQUASHED IMAGE: 400Mi DOWNLOAD ON LAYER CHANGE: 400Mi