ContaoConference23: The Anatomy Of A Docker Image

Over the past few years, container technologies (like Docker) have become essential to many a developer's workflow -- be it for local development, testing, CI or production rollouts. In this talk, I will shed light on one of the underlying principles of Docker: the OCI image and distribution specifications. These describe the file format of a Docker image and the API of a Docker registry. Understanding these will help you understand what's actually going on when you run commands like "docker build" or "docker pull". I will also present a case study of a project in which we used these specifications to build a container image solution that implemented features that would have been impossible to implement using the regular tools.

Transcript

1. Victoire Joncheray https://unsplash.com/photos/blue-intermodal-container-XsP7GCLMWjM ANATOMY THE OF A DOCKER IMAGE Contao Conference, October 19th, 2023 MARTIN HELMICH @mittwald
2. MARTIN HELMICH Head of Architecture & Chief Tech Evangelist Lecturer, Software Engineering & Cloud Computing Sci-Fi-Nerd, Metalhead, Amateur Woodworker
4. CONTAINERS ARE EVERYWHERE Niklas9416, https://pixabay.com/de/photos/hamburg-hafen-schiff-container-6849995/
5. PROTO TYPING DEV TEST DEPLOY OPS
6. INFRASTRUCTURE YOUR STUFF ? YOUR PROBLEM ALSO YOUR PROBLEM
7. PROD PREPROD ACCEPTANCE INTEGRATION TEST ONE APP MANY ENVIRONMENTS DEVELOPER ALICE DEVELOPER BOB DEVELOPER CHARLIE YOUR STUFF
8. WORKS ON MY MACHINE pen_ash https://pixabay.com/photos/ostrich-flightless-bird-bird-avian-4366263/
9. pen_ash https://pixabay.com/photos/ostrich-flightless-bird-bird-avian-4366263/ LOOKS LIKE YOU’RE SUFFERING FROM AN EXTREME CASE OF… ENVIRONMENT DISPARITY
10. INFRASTRUCTURE YOUR STUFF ! YOUR PROBLEM YOUR PROBLEM NOT CONTAINER IMAGE YOUR STUFF REQUIRED ENVIRONMENT
11. MY PROBLEM kirill_makes_pics https://pixabay.com/photos/despaired-businessman-business-2261021/
12. As a web project owner, I can freely configure system components in my environment, so that I can operate a wide range of different applications with different system requirements.
13. > ID=$(mw app install contao --version=^4.13 -q) > mw app dependency update $ID \ --set 'php=^8.2' \ --set 'graphicsmagick=*' \ --set 'nodejs=^20' TASK: BUILD A CONTAINER IMAGE FROM THIS USER-PROVIDED SPECIFICATION UNRELATED: https://github.com/mittwald/cli
14. FROM ubuntu:22.04 RUN apt-get update -y RUN apt-get install –y php RUN apt-get install –y graphicsmagick RUN apt-get install –y nodejs INTRODUCING THE CONTAINERFILE
15. FROM ubuntu:22.04 RUN apt-get update -y {{- range $software, $version := .UserRequirements }} RUN apt-get install -y {{ $software }}={{ $version }} {{- end }} IS THIS A GOOD IDEA? (spoiler alert: it is, indeed, not) (actually, WAY more complicated)
16. php=^8.2 graphicsmagick=* nodejs=^20 FROM ubuntu 22.04 {{- range $sw, $v := .User apt-get install -y {{ $sw {{- end }} USER ENVIRONMENT SPECIFICATION TEMPLATED CONTAINERFILE (actually, WAY more complicated) CONTAINER IMAGE DOCKER BUILD HARBOR DOCKER PUSH DOCKER PULL CONTAINER REGISTRY USER PROFIT 💰
17. > ID=$(mw app install contao --version=^4.13 -q) > mw app dependency update $ID \ --set 'php=^8.2' \ --set 'graphicsmagick=*' \ --set 'nodejs=^20' ⏳ updating environment… ✅ environment updated after 27m51s 🫠
18. kirill_makes_pics https://pixabay.com/photos/despaired-businessman-business-2261021/
19. COMMON CONTAINER BUILD OPTIMIZATION TARGETS 📐 Image size ⏱ Build Time 🏎 Download time yellowcat https://pixabay.com/photos/sport-clock-jogging-time-fitness-3874743/
20. MEH. WHO CARES? 😐
21. CONTAINER RUNTIME PULL / RUN BUILD ENVIRONMENT CONTAINER REGISTRY PUSH BUILD IMAGE SPEC https://github.com/opencontainers/image- spec DISTRIBUTION SPEC https://github.com/opencontainers/distribution- spec RUNTIME SPEC https://github.com/opencontainers/runtime- spec
23. file1 file2 file3 file2 file4 FILESYSTEM LAYERS UNION MOUNT (OVERLAYFS or AUFS) file1 file2 file3 file4
24. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y composer php-fpm COPY composer.json src /var/www/html/ RUN composer install --no-dev # to be continued… CONTAINERFILE
25. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y composer php-fpm COPY composer.json src /var/www/html/ RUN composer install --no-dev # to be continued… CACHED UNCACHED A change invalidates all cached layers after this one PRO TIP Pull your base image before building to make sure it’s current #security PRO TIP Most of the time, you don’t need dev dependencies in your container image.
26. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y composer php-fpm COPY composer.json src /var/www/html/ RUN composer install --no-dev # to be continued… This file changes (presumably) relatively frequently This file changes (presumably) relatively infrequently This step needs only to run when the composer.json changed
27. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y composer php-fpm COPY composer.json /var/www/html/ RUN composer install --no-dev COPY src /var/www/html/ # to be continued… CACHED UNCACHED Build instruction order determines cache utilization!
28. FROM ubuntu:22.04 AS builder RUN apt-get update –y RUN apt-get install –y composer COPY composer.json app/ RUN composer install --no-dev FROM ubuntu:22.04 RUN apt-get update –y && apt-get install –y php-fpm COPY --from=builder /var/www/html/vendor ↵ /var/www/html/ COPY src /var/www/html/ # to be continued… CACHED UNCACHED Use multistage builds to optimize even further CACHED
29. composer.json auth.json vendor/ auth.json COPY composer.json auth.json /app RUN composer install --no-dev RUN rm auth.json composer.json auth.json vendor/ DON’T DO THIS! Your credentials are still there!
30. composer.json auth.json vendor/ COPY composer.json auth.json /app RUN composer install --no-dev vendor/ DO THIS, instead And keep your credentials safe! FROM <base> AS builder FROM <base> vendor/ COPY --from=base vendor vendor
31. REGULAR DOCKER USERS ME https://knowyourmeme.com/memes/two-guys-on-a-bus
32. CONFIGURATION (JSON) sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744 FS LAYERS sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7 sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4 sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9 sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216 sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8 sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476 sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736 sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8 sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9 (USUAL FILE FORMATS: TAR or TAR.GZ) CONTENTS OF A DOCKER IMAGE MANIFEST
33. INDEX (JSON) amd64 armv7 arm64 MANIFEST CONFIGURATION (JSON) sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744 FS LAYERS sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7 sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4 sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9 sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216 sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8 sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476 sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736 sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8 sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9 (USUAL FILE FORMATS: TAR or TAR.GZ)
34. amd64 armv7 arm64 GET /v2/{image}/manifests/{tag} Accept: application/vnd.oci.image.index.v1+json { "mediaType": "application/vnd.oci.image.index.v1+json", "schemaVersion": 2, "manifests": [ { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:f10dac435406078cfc491daa0cd99598b10909ae259a4e13d…", "size": 3145, "platform": { "architecture": "arm", "os": "linux", "variant": "v7" } }, […] ] } INDEX (JSON)
35. GET /v2/{image}/manifests/{digest} Accept: application/vnd.oci.image.manifest.v1+json { "mediaType": "application/vnd.oci.image.manifest.v1+json", "schemaVersion": 2, "config": { "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:966da77cd61840d8d91d34b9cbdae2104683f93be5550f9819b…", "size": 14426 }, "layers": [ { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:faef57eae888cbe4a5613eca6741b5e48d768b83f6088858a…", "size": 29124829 }, […] ] } MANIFEST ION 63d1504addbe7eb6b82e7798fb744 S 6b87074e55fea6bd1ecae066693c7 13d066d60543ac1b176802dd8638d 6e956b4c0bd9f0c29aaf1b9c5b49b 9716aa4f8df2d148dc1f947b1abe4 995ba25193b0f43ba8bbc4abd99f9 0daf28b405740eb498f9c07837f4f 007953eb385eb7c8425461fb2a54f a3ede53659006c66744b472a95216 367ede931b8e2a5272c562cdcfee8 7e60cb9b50efb1ead4d05d35f079f 343953d068478845c22870c596476 f5e07dc2fd02f0c1b5931ba2b230d 960290328e8829987721c61143736 f9b1e9aee0d51a5320cd5ba6d60a8 f578a8647c7811807d0a2723916b9 R or TAR.GZ)
36. “IT’S JUST A REST API, ISN’T IT?” r.HandleFunc( "/v2/{repo}/{image}/manifests/{tag}", func(w http.ResponseWriter, r *http.Response) { repo, image, tag := manifestRequestVars(r) if manifest, ok := GetManifest(repo, image, tag); ok { json.NewEncoder(w).Write(manifest) } }, ) r.HandleFunc( "/v2/{repo}/{image}/blobs/{digest}", func(w http.ResponseWriter, r *http.Response) { repo, image, digest := blobRequestVars(r) file, err := OpenBlob(repo, image, digest) if err != nil { // handle error } io.Copy(file, r) }, )
37. php=^8.2 graphicsmagick=* nodejs=^20 USER ENVIRONMENT SPECIFICATION PRE-PACKAGED, INDEPENDENT (!) LAYERS (VIRTUAL) CONTAINER IMAGE GENERATE IMAGE MANIFEST CUSTOM REGISTRY SERVE VIA CUSTOM API DOCKER PULL CONTAINER REGISTRY PROFIT 💰 USER BUILT AHEAD-OF-TIME
38. ThMilherou https://pixabay.com/photos/workshop-carpenter-crafts-artisan-4863393/ TOOL RECOMMENDATIONS
39. DIVE (https://github.com/wagoodman/dive) Explore individual layers in a given image
40. SKOPEO (https://github.com/containers/skopeo) Inspect (and manipulate) images in remote registries
41. SLIM TOOLKIT (https://github.com/slimtoolkit/slim) Optimize images for size, using black magic (requires caution!)
42. HOOKED ON CONTAINERS? https://github.com/mittwald/feature-requests/issues/1
43. END THE Three-shots https://pixabay.com/photos/ocean-blue-whale-sea-wave-whale-2051760/
44. BONUS SLIDES
45. FROM big-ass-image:v1.2.3 AS source FROM scratch COPY --from=source / / # NOTE: Make sure to set environment variables CMD … ENTRYPOINT … SQUASHING IMAGES
46. SQUASHING IMAGES ORIGINAL IMAGE: 1Gi SQUASHED IMAGE: 400Mi
47. SQUASHING IMAGES ORIGINAL IMAGE: 1Gi DOWNLOAD ON LAYER CHANGE: 10Mi SQUASHED IMAGE: 400Mi DOWNLOAD ON LAYER CHANGE: 400Mi
48. DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION ON PREMISE IaaS PaaS YOUR PROBLEM NOT YOUR PROBLEM MORE CONTROL MORE PROBLEMS LESS CONTROL LESS PROBLEMS
49. DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION ON PREMISE DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION IaaS DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION PaaS YOUR PROBLEM NOT YOUR PROBLEM CaaS DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS CONTAINER RUNTIME CONTAINER OS RUNTIME APPLICATION MORE CONTROL MORE PROBLEMS LESS CONTROL LESS PROBLEMS