T3DD23: The Anatomy Of A Docker Image

Over the past few years, container technologies (like Docker) have become essential to many a developer's workflow -- be it for local development, testing, CI or production rollouts. In this talk, I will shed light on one of the underlying principles of Docker: the OCI image and distribution specifications. These describe the file format of a Docker image and the API of a Docker registry. Understanding these will help you understand what's actually going on when you run commands like "docker build" or "docker pull". I will also present a case study of a project in which we used these specifications to build a container image solution that implemented features that would have been impossible to implement using the regular tools.

Transcript

1. Lucas Hoang
   https://unsplash.com/photos/RJPEfLuv6Ao
   ANATOMY
   THE
   OF A
   DOCKER IMAGE
   TYPO3 Developer Days, August 4th, 2023
   MARTIN HELMICH @mittwald
   

   View Slide

2. MARTIN HELMICH
   Head of Architecture &
   Chief Tech Evangelist
   Lecturer, Software Engineering
   & Cloud Computing
   Sci-Fi-Nerd, Metalhead,
   Amateur Woodworker
   

   View Slide

3. CONTAINERS ARE
   EVERYWHERE
   Niklas9416,
   https://pixabay.com/de/photos/hamburg-hafen-schiff-container-6849995/
   

   View Slide

4. PROTO
   TYPING
   DEV TEST DEPLOY OPS
   

   View Slide

5. MY
   PROBLEM
   kirill_makes_pics
   https://pixabay.com/photos/despaired-businessman-business-2261021/
   

   View Slide

6. As a web project owner, I can freely
   configure system components in
   my environment, so that I can
   operate a wide range of different
   applications with different system
   requirements.
   

   View Slide

7. > mw project create -q --update-context --wait
   > ID=$(mw app install typo3 --version=12.4.4 -q --wait)
   > mw app dependency update $ID \
   --set 'php=^8.2' \
   --set 'graphicsmagick=*' \
   --set 'nodejs=^20'
   TASK:
   BUILD A CONTAINER IMAGE
   FROM THIS USER-PROVIDED SPECIFICATION
   UNRELATED: https://github.com/mittwald/cli
   

   View Slide

8. FROM ubuntu:22.04
   RUN apt-get update -y
   RUN apt-get install –y php
   RUN apt-get install –y graphicsmagick
   RUN apt-get install –y nodejs
   INTRODUCING THE
   CONTAINERFILE
   

   View Slide

9. FROM ubuntu:22.04
   RUN apt-get update -y
   {{- range $software, $version := .UserRequirements }}
   RUN apt-get install -y {{ $software }}={{ $version }}
   {{- end }}
   IS THIS A GOOD IDEA?
   (spoiler alert: it is, indeed, not)
   (actually, WAY more complicated)
   

   View Slide

10. php=^8.2
    graphicsmagick=*
    nodejs=^20
    FROM ubuntu 22.04
    {{- range $sw, $v := .User
    apt-get install -y {{ $sw
    {{- end }}
    USER ENVIRONMENT
    SPECIFICATION
    TEMPLATED
    CONTAINERFILE
    (actually, WAY more complicated)
    CONTAINER
    IMAGE
    DOCKER BUILD
    HARBOR
    DOCKER
    PUSH
    DOCKER
    PULL
    CONTAINER
    REGISTRY
    USER
    PROFIT
    💰
    

    View Slide

11. > mw project create -q --update-context --wait
    > ID=$(mw app install typo3 --version=12.4.4 -q --wait)
    > mw app dependency update $ID \
    --set 'php=^8.2' \
    --set 'graphicsmagick=*' \
    --set 'nodejs=^20‘
    ⏳ updating environment…
    ✅ environment updated after 27m51s
    UNRELATED: https://github.com/mittwald/cli
    🫠
    

    View Slide

12. kirill_makes_pics
    https://pixabay.com/photos/despaired-businessman-business-2261021/
    

    View Slide

13. COMMON
    CONTAINER BUILD
    OPTIMIZATION
    TARGETS
    📐 Image size
    ⏱ Build Time
    🏎 Download time
    yellowcat
    https://pixabay.com/photos/sport-clock-jogging-time-fitness-3874743/
    

    View Slide

14. MEH. WHO CARES? 😐
    

    View Slide

15. LIFECYCLE
    OF A CONTAINER
    PULL / RUN
    PUSH
    BUILD
    

    View Slide

16. CONTAINER
    RUNTIME
    PULL / RUN
    BUILD
    ENVIRONMENT
    CONTAINER
    REGISTRY
    PUSH
    BUILD IMAGE SPEC
    https://github.com/opencontainers/image-spec
    DISTRIBUTION SPEC
    https://github.com/opencontainers/distribution-spec
    RUNTIME SPEC
    https://github.com/opencontainers/runtime-spec
    

    View Slide

17. View Slide

18. file1 file2
    file3
    file2 file4
    FILESYSTEM
    LAYERS
    UNION MOUNT
    (OVERLAYFS or AUFS) file1 file2 file3 file4
    

    View Slide

19. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json src /var/www/html/
    RUN composer install --no-dev
    # to be continued…
    CONTAINERFILE
    

    View Slide

20. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json src /var/www/html/
    RUN composer install --no-dev
    # to be continued…
    CACHED
    UNCACHED
    A change invalidates all
    cached layers after this one
    PRO TIP Pull your base image before
    building to make sure it’s current #security
    PRO TIP Most of the time, you
    don’t need dev dependencies
    in your container image.
    

    View Slide

21. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json src /var/www/html/
    RUN composer install --no-dev
    # to be continued… This file changes (presumably)
    relatively frequently
    This file changes
    (presumably)
    relatively infrequently
    This step needs only to run
    when the composer.json
    changed
    

    View Slide

22. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json /var/www/html/
    RUN composer install --no-dev
    COPY src /var/www/html/
    # to be continued…
    CACHED
    UNCACHED
    Build instruction order
    determines cache utilization!
    

    View Slide

23. FROM ubuntu:22.04 AS builder
    RUN apt-get update –y
    RUN apt-get install –y composer
    COPY composer.json app/
    RUN composer install --no-dev
    FROM ubuntu:22.04
    RUN apt-get update –y && apt-get install –y php-fpm
    COPY --from=builder /var/www/html/vendor ↵
    /var/www/html/
    COPY src /var/www/html/
    # to be continued…
    CACHED
    UNCACHED
    Use multistage builds to
    optimize even further
    CACHED
    

    View Slide

24. composer.json auth.json
    vendor/
    auth.json
    COPY composer.json auth.json /app
    RUN composer install --no-dev
    RUN rm auth.json
    composer.json auth.json vendor/
    DON’T
    DO THIS!
    Your credentials are still there!
    

    View Slide

25. composer.json auth.json
    vendor/
    COPY composer.json auth.json /app
    RUN composer install --no-dev
    vendor/
    DO
    THIS, instead
    And keep your credentials safe!
    FROM AS builder
    FROM
    vendor/
    COPY --from=base vendor vendor
    

    View Slide

26. REGULAR DOCKER
    USERS
    ME
    https://knowyourmeme.com/memes/two-guys-on-a-bus
    

    View Slide

27. CONFIGURATION (JSON)
    sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744
    FS LAYERS
    sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7
    sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d
    sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b
    sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4
    sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9
    sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f
    sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f
    sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216
    sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8
    sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f
    sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476
    sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d
    sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736
    sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8
    sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9
    (USUAL FILE FORMATS: TAR or TAR.GZ)
    CONTENTS
    OF A
    DOCKER
    IMAGE
    MANIFEST
    

    View Slide

28. INDEX
    (JSON)
    amd64
    armv7
    arm64
    MANIFEST
    CONFIGURATION (JSON)
    sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744
    FS LAYERS
    sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7
    sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d
    sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b
    sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4
    sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9
    sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f
    sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f
    sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216
    sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8
    sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f
    sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476
    sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d
    sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736
    sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8
    sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9
    (USUAL FILE FORMATS: TAR or TAR.GZ)
    

    View Slide

29. amd64
    armv7
    arm64
    GET /v2/{image}/manifests/{tag}
    Accept: application/vnd.oci.image.index.v1+json
    {
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "schemaVersion": 2,
    "manifests": [
    {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:f10dac435406078cfc491daa0cd99598b10909ae259a4e13d…",
    "size": 3145,
    "platform": {
    "architecture": "arm",
    "os": "linux",
    "variant": "v7"
    }
    },
    […]
    ]
    }
    INDEX
    (JSON)
    

    View Slide

30. GET /v2/{image}/manifests/{digest}
    Accept: application/vnd.oci.image.manifest.v1+json
    {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "schemaVersion": 2,
    "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:966da77cd61840d8d91d34b9cbdae2104683f93be5550f9819b…",
    "size": 14426
    },
    "layers": [
    {
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
    "digest": "sha256:faef57eae888cbe4a5613eca6741b5e48d768b83f6088858a…",
    "size": 29124829
    },
    […]
    ]
    }
    MANIFEST
    ION
    63d1504addbe7eb6b82e7798fb744
    S
    6b87074e55fea6bd1ecae066693c7
    13d066d60543ac1b176802dd8638d
    6e956b4c0bd9f0c29aaf1b9c5b49b
    9716aa4f8df2d148dc1f947b1abe4
    995ba25193b0f43ba8bbc4abd99f9
    0daf28b405740eb498f9c07837f4f
    007953eb385eb7c8425461fb2a54f
    a3ede53659006c66744b472a95216
    367ede931b8e2a5272c562cdcfee8
    7e60cb9b50efb1ead4d05d35f079f
    343953d068478845c22870c596476
    f5e07dc2fd02f0c1b5931ba2b230d
    960290328e8829987721c61143736
    f9b1e9aee0d51a5320cd5ba6d60a8
    f578a8647c7811807d0a2723916b9
    R or TAR.GZ)
    

    View Slide

31. “IT’S JUST A REST API,
    ISN’T IT?”
    r.HandleFunc(
    "/v2/{repo}/{image}/manifests/{tag}",
    func(w http.ResponseWriter, r *http.Response) {
    repo, image, tag := manifestRequestVars(r)
    if manifest, ok := GetManifest(repo, image, tag); ok {
    json.NewEncoder(w).Write(manifest)
    }
    },
    )
    r.HandleFunc(
    "/v2/{repo}/{image}/blobs/{digest}",
    func(w http.ResponseWriter, r *http.Response) {
    repo, image, digest := blobRequestVars(r)
    file, err := OpenBlob(repo, image, digest)
    if err != nil {
    // handle error
    }
    io.Copy(file, r)
    },
    )
    

    View Slide

32. php=^8.2
    graphicsmagick=*
    nodejs=^20
    USER ENVIRONMENT
    SPECIFICATION
    PRE-PACKAGED,
    INDEPENDENT (!) LAYERS
    (VIRTUAL)
    CONTAINER
    IMAGE
    GENERATE
    IMAGE MANIFEST
    CUSTOM
    REGISTRY
    SERVE VIA
    CUSTOM API
    DOCKER
    PULL
    CONTAINER
    REGISTRY
    PROFIT
    💰
    USER
    BUILT
    AHEAD-OF-TIME
    

    View Slide

33. > mw project create -q --update-context --wait
    > ID=$(mw app install typo3 --version=12.4.4 -q --wait)
    > mw app dependency update $ID \
    --set 'php=^8.2' \
    --set 'graphicsmagick=*' \
    --set 'nodejs=^20‘
    ⏳ updating environment…
    ✅ environment updated after 180ms
    UNRELATED: https://github.com/mittwald/cli
    🚀
    (APPROXIMATELY)
    

    View Slide

34. ThMilherou
    https://pixabay.com/photos/workshop-carpenter-crafts-artisan-4863393/
    TOOL
    RECOMMENDATIONS
    

    View Slide

35. DIVE (https://github.com/wagoodman/dive)
    Explore individual layers in a given image
    

    View Slide

36. SKOPEO (https://github.com/containers/skopeo)
    Inspect (and manipulate) images in remote registries
    

    View Slide

37. SLIM TOOLKIT (https://github.com/slimtoolkit/slim)
    Optimize images for size, using black magic (requires caution!)
    

    View Slide

38. 
    

    View Slide

39. END
    THE
    Three-shots
    https://pixabay.com/photos/ocean-blue-whale-sea-wave-whale-2051760/
    

    View Slide

40. BONUS
    SLIDES
    

    View Slide

41. FROM big-ass-image:v1.2.3 AS source
    FROM scratch
    COPY --from=source / /
    # NOTE: Make sure to set environment variables
    CMD …
    ENTRYPOINT …
    SQUASHING
    IMAGES
    

    View Slide

42. SQUASHING
    IMAGES
    ORIGINAL IMAGE: 1Gi
    SQUASHED IMAGE: 400Mi
    

    View Slide

43. SQUASHING
    IMAGES
    ORIGINAL IMAGE: 1Gi
    DOWNLOAD ON LAYER CHANGE: 10Mi
    SQUASHED IMAGE: 400Mi
    DOWNLOAD ON LAYER CHANGE: 400Mi
    

    View Slide