$30 off During Our Annual Pro Sale. View Details »

T3DD23: The Anatomy Of A Docker Image

T3DD23: The Anatomy Of A Docker Image

Over the past few years, container technologies (like Docker) have become essential to many a developer's workflow -- be it for local development, testing, CI or production rollouts. In this talk, I will shed light on one of the underlying principles of Docker: the OCI image and distribution specifications. These describe the file format of a Docker image and the API of a Docker registry. Understanding these will help you understand what's actually going on when you run commands like "docker build" or "docker pull". I will also present a case study of a project in which we used these specifications to build a container image solution that implemented features that would have been impossible to implement using the regular tools.

Martin Helmich

August 04, 2023
Tweet

More Decks by Martin Helmich

Other Decks in Technology

Transcript

  1. Lucas Hoang
    https://unsplash.com/photos/RJPEfLuv6Ao
    ANATOMY
    THE
    OF A
    DOCKER IMAGE
    TYPO3 Developer Days, August 4th, 2023
    MARTIN HELMICH @mittwald

    View Slide

  2. MARTIN HELMICH
    Head of Architecture &
    Chief Tech Evangelist
    Lecturer, Software Engineering
    & Cloud Computing
    Sci-Fi-Nerd, Metalhead,
    Amateur Woodworker

    View Slide

  3. CONTAINERS ARE
    EVERYWHERE
    Niklas9416,
    https://pixabay.com/de/photos/hamburg-hafen-schiff-container-6849995/

    View Slide

  4. PROTO
    TYPING
    DEV TEST DEPLOY OPS

    View Slide

  5. MY
    PROBLEM
    kirill_makes_pics
    https://pixabay.com/photos/despaired-businessman-business-2261021/

    View Slide

  6. As a web project owner, I can freely
    configure system components in
    my environment, so that I can
    operate a wide range of different
    applications with different system
    requirements.

    View Slide

  7. > mw project create -q --update-context --wait
    > ID=$(mw app install typo3 --version=12.4.4 -q --wait)
    > mw app dependency update $ID \
    --set 'php=^8.2' \
    --set 'graphicsmagick=*' \
    --set 'nodejs=^20'
    TASK:
    BUILD A CONTAINER IMAGE
    FROM THIS USER-PROVIDED SPECIFICATION
    UNRELATED: https://github.com/mittwald/cli

    View Slide

  8. FROM ubuntu:22.04
    RUN apt-get update -y
    RUN apt-get install –y php
    RUN apt-get install –y graphicsmagick
    RUN apt-get install –y nodejs
    INTRODUCING THE
    CONTAINERFILE

    View Slide

  9. FROM ubuntu:22.04
    RUN apt-get update -y
    {{- range $software, $version := .UserRequirements }}
    RUN apt-get install -y {{ $software }}={{ $version }}
    {{- end }}
    IS THIS A GOOD IDEA?
    (spoiler alert: it is, indeed, not)
    (actually, WAY more complicated)

    View Slide

  10. php=^8.2
    graphicsmagick=*
    nodejs=^20
    FROM ubuntu 22.04
    {{- range $sw, $v := .User
    apt-get install -y {{ $sw
    {{- end }}
    USER ENVIRONMENT
    SPECIFICATION
    TEMPLATED
    CONTAINERFILE
    (actually, WAY more complicated)
    CONTAINER
    IMAGE
    DOCKER BUILD
    HARBOR
    DOCKER
    PUSH
    DOCKER
    PULL
    CONTAINER
    REGISTRY
    USER
    PROFIT
    💰

    View Slide

  11. > mw project create -q --update-context --wait
    > ID=$(mw app install typo3 --version=12.4.4 -q --wait)
    > mw app dependency update $ID \
    --set 'php=^8.2' \
    --set 'graphicsmagick=*' \
    --set 'nodejs=^20‘
    ⏳ updating environment…
    ✅ environment updated after 27m51s
    UNRELATED: https://github.com/mittwald/cli
    🫠

    View Slide

  12. kirill_makes_pics
    https://pixabay.com/photos/despaired-businessman-business-2261021/

    View Slide

  13. COMMON
    CONTAINER BUILD
    OPTIMIZATION
    TARGETS
    📐 Image size
    ⏱ Build Time
    🏎 Download time
    yellowcat
    https://pixabay.com/photos/sport-clock-jogging-time-fitness-3874743/

    View Slide

  14. MEH. WHO CARES? 😐

    View Slide

  15. LIFECYCLE
    OF A CONTAINER
    PULL / RUN
    PUSH
    BUILD

    View Slide

  16. CONTAINER
    RUNTIME
    PULL / RUN
    BUILD
    ENVIRONMENT
    CONTAINER
    REGISTRY
    PUSH
    BUILD IMAGE SPEC
    https://github.com/opencontainers/image-spec
    DISTRIBUTION SPEC
    https://github.com/opencontainers/distribution-spec
    RUNTIME SPEC
    https://github.com/opencontainers/runtime-spec

    View Slide

  17. View Slide

  18. file1 file2
    file3
    file2 file4
    FILESYSTEM
    LAYERS
    UNION MOUNT
    (OVERLAYFS or AUFS) file1 file2 file3 file4

    View Slide

  19. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json src /var/www/html/
    RUN composer install --no-dev
    # to be continued…
    CONTAINERFILE

    View Slide

  20. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json src /var/www/html/
    RUN composer install --no-dev
    # to be continued…
    CACHED
    UNCACHED
    A change invalidates all
    cached layers after this one
    PRO TIP Pull your base image before
    building to make sure it’s current #security
    PRO TIP Most of the time, you
    don’t need dev dependencies
    in your container image.

    View Slide

  21. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json src /var/www/html/
    RUN composer install --no-dev
    # to be continued… This file changes (presumably)
    relatively frequently
    This file changes
    (presumably)
    relatively infrequently
    This step needs only to run
    when the composer.json
    changed

    View Slide

  22. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json /var/www/html/
    RUN composer install --no-dev
    COPY src /var/www/html/
    # to be continued…
    CACHED
    UNCACHED
    Build instruction order
    determines cache utilization!

    View Slide

  23. FROM ubuntu:22.04 AS builder
    RUN apt-get update –y
    RUN apt-get install –y composer
    COPY composer.json app/
    RUN composer install --no-dev
    FROM ubuntu:22.04
    RUN apt-get update –y && apt-get install –y php-fpm
    COPY --from=builder /var/www/html/vendor ↵
    /var/www/html/
    COPY src /var/www/html/
    # to be continued…
    CACHED
    UNCACHED
    Use multistage builds to
    optimize even further
    CACHED

    View Slide

  24. composer.json auth.json
    vendor/
    auth.json
    COPY composer.json auth.json /app
    RUN composer install --no-dev
    RUN rm auth.json
    composer.json auth.json vendor/
    DON’T
    DO THIS!
    Your credentials are still there!

    View Slide

  25. composer.json auth.json
    vendor/
    COPY composer.json auth.json /app
    RUN composer install --no-dev
    vendor/
    DO
    THIS, instead
    And keep your credentials safe!
    FROM AS builder
    FROM
    vendor/
    COPY --from=base vendor vendor

    View Slide

  26. REGULAR DOCKER
    USERS
    ME
    https://knowyourmeme.com/memes/two-guys-on-a-bus

    View Slide

  27. CONFIGURATION (JSON)
    sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744
    FS LAYERS
    sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7
    sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d
    sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b
    sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4
    sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9
    sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f
    sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f
    sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216
    sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8
    sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f
    sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476
    sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d
    sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736
    sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8
    sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9
    (USUAL FILE FORMATS: TAR or TAR.GZ)
    CONTENTS
    OF A
    DOCKER
    IMAGE
    MANIFEST

    View Slide

  28. INDEX
    (JSON)
    amd64
    armv7
    arm64
    MANIFEST
    CONFIGURATION (JSON)
    sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744
    FS LAYERS
    sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7
    sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d
    sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b
    sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4
    sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9
    sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f
    sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f
    sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216
    sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8
    sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f
    sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476
    sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d
    sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736
    sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8
    sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9
    (USUAL FILE FORMATS: TAR or TAR.GZ)

    View Slide

  29. amd64
    armv7
    arm64
    GET /v2/{image}/manifests/{tag}
    Accept: application/vnd.oci.image.index.v1+json
    {
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "schemaVersion": 2,
    "manifests": [
    {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:f10dac435406078cfc491daa0cd99598b10909ae259a4e13d…",
    "size": 3145,
    "platform": {
    "architecture": "arm",
    "os": "linux",
    "variant": "v7"
    }
    },
    […]
    ]
    }
    INDEX
    (JSON)

    View Slide

  30. GET /v2/{image}/manifests/{digest}
    Accept: application/vnd.oci.image.manifest.v1+json
    {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "schemaVersion": 2,
    "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:966da77cd61840d8d91d34b9cbdae2104683f93be5550f9819b…",
    "size": 14426
    },
    "layers": [
    {
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
    "digest": "sha256:faef57eae888cbe4a5613eca6741b5e48d768b83f6088858a…",
    "size": 29124829
    },
    […]
    ]
    }
    MANIFEST
    ION
    63d1504addbe7eb6b82e7798fb744
    S
    6b87074e55fea6bd1ecae066693c7
    13d066d60543ac1b176802dd8638d
    6e956b4c0bd9f0c29aaf1b9c5b49b
    9716aa4f8df2d148dc1f947b1abe4
    995ba25193b0f43ba8bbc4abd99f9
    0daf28b405740eb498f9c07837f4f
    007953eb385eb7c8425461fb2a54f
    a3ede53659006c66744b472a95216
    367ede931b8e2a5272c562cdcfee8
    7e60cb9b50efb1ead4d05d35f079f
    343953d068478845c22870c596476
    f5e07dc2fd02f0c1b5931ba2b230d
    960290328e8829987721c61143736
    f9b1e9aee0d51a5320cd5ba6d60a8
    f578a8647c7811807d0a2723916b9
    R or TAR.GZ)

    View Slide

  31. “IT’S JUST A REST API,
    ISN’T IT?”
    r.HandleFunc(
    "/v2/{repo}/{image}/manifests/{tag}",
    func(w http.ResponseWriter, r *http.Response) {
    repo, image, tag := manifestRequestVars(r)
    if manifest, ok := GetManifest(repo, image, tag); ok {
    json.NewEncoder(w).Write(manifest)
    }
    },
    )
    r.HandleFunc(
    "/v2/{repo}/{image}/blobs/{digest}",
    func(w http.ResponseWriter, r *http.Response) {
    repo, image, digest := blobRequestVars(r)
    file, err := OpenBlob(repo, image, digest)
    if err != nil {
    // handle error
    }
    io.Copy(file, r)
    },
    )

    View Slide

  32. php=^8.2
    graphicsmagick=*
    nodejs=^20
    USER ENVIRONMENT
    SPECIFICATION
    PRE-PACKAGED,
    INDEPENDENT (!) LAYERS
    (VIRTUAL)
    CONTAINER
    IMAGE
    GENERATE
    IMAGE MANIFEST
    CUSTOM
    REGISTRY
    SERVE VIA
    CUSTOM API
    DOCKER
    PULL
    CONTAINER
    REGISTRY
    PROFIT
    💰
    USER
    BUILT
    AHEAD-OF-TIME

    View Slide

  33. > mw project create -q --update-context --wait
    > ID=$(mw app install typo3 --version=12.4.4 -q --wait)
    > mw app dependency update $ID \
    --set 'php=^8.2' \
    --set 'graphicsmagick=*' \
    --set 'nodejs=^20‘
    ⏳ updating environment…
    ✅ environment updated after 180ms
    UNRELATED: https://github.com/mittwald/cli
    🚀
    (APPROXIMATELY)

    View Slide

  34. ThMilherou
    https://pixabay.com/photos/workshop-carpenter-crafts-artisan-4863393/
    TOOL
    RECOMMENDATIONS

    View Slide

  35. DIVE (https://github.com/wagoodman/dive)
    Explore individual layers in a given image

    View Slide

  36. SKOPEO (https://github.com/containers/skopeo)
    Inspect (and manipulate) images in remote registries

    View Slide

  37. SLIM TOOLKIT (https://github.com/slimtoolkit/slim)
    Optimize images for size, using black magic (requires caution!)

    View Slide


  38. View Slide

  39. END
    THE
    Three-shots
    https://pixabay.com/photos/ocean-blue-whale-sea-wave-whale-2051760/

    View Slide

  40. BONUS
    SLIDES

    View Slide

  41. FROM big-ass-image:v1.2.3 AS source
    FROM scratch
    COPY --from=source / /
    # NOTE: Make sure to set environment variables
    CMD …
    ENTRYPOINT …
    SQUASHING
    IMAGES

    View Slide

  42. SQUASHING
    IMAGES
    ORIGINAL IMAGE: 1Gi
    SQUASHED IMAGE: 400Mi

    View Slide

  43. SQUASHING
    IMAGES
    ORIGINAL IMAGE: 1Gi
    DOWNLOAD ON LAYER CHANGE: 10Mi
    SQUASHED IMAGE: 400Mi
    DOWNLOAD ON LAYER CHANGE: 400Mi

    View Slide