$30 off During Our Annual Pro Sale. View Details »

ContaoConference23: The Anatomy Of A Docker Image

ContaoConference23: The Anatomy Of A Docker Image

Over the past few years, container technologies (like Docker) have become essential to many a developer's workflow -- be it for local development, testing, CI or production rollouts. In this talk, I will shed light on one of the underlying principles of Docker: the OCI image and distribution specifications. These describe the file format of a Docker image and the API of a Docker registry. Understanding these will help you understand what's actually going on when you run commands like "docker build" or "docker pull". I will also present a case study of a project in which we used these specifications to build a container image solution that implemented features that would have been impossible to implement using the regular tools.

Martin Helmich

October 24, 2023
Tweet

More Decks by Martin Helmich

Other Decks in Technology

Transcript

  1. Victoire Joncheray
    https://unsplash.com/photos/blue-intermodal-container-XsP7GCLMWjM
    ANATOMY
    THE
    OF A
    DOCKER IMAGE
    Contao Conference, October 19th, 2023
    MARTIN HELMICH @mittwald

    View Slide

  2. MARTIN HELMICH
    Head of Architecture &
    Chief Tech Evangelist
    Lecturer, Software Engineering
    & Cloud Computing
    Sci-Fi-Nerd, Metalhead,
    Amateur Woodworker

    View Slide

  3. View Slide

  4. CONTAINERS ARE
    EVERYWHERE
    Niklas9416,
    https://pixabay.com/de/photos/hamburg-hafen-schiff-container-6849995/

    View Slide

  5. PROTO
    TYPING
    DEV TEST DEPLOY OPS

    View Slide

  6. INFRASTRUCTURE
    YOUR
    STUFF
    ?
    YOUR PROBLEM
    ALSO
    YOUR PROBLEM

    View Slide

  7. PROD
    PREPROD
    ACCEPTANCE
    INTEGRATION
    TEST
    ONE APP
    MANY ENVIRONMENTS
    DEVELOPER
    ALICE
    DEVELOPER
    BOB
    DEVELOPER
    CHARLIE
    YOUR
    STUFF

    View Slide

  8. WORKS
    ON MY
    MACHINE
    pen_ash
    https://pixabay.com/photos/ostrich-flightless-bird-bird-avian-4366263/

    View Slide

  9. pen_ash
    https://pixabay.com/photos/ostrich-flightless-bird-bird-avian-4366263/
    LOOKS LIKE YOU’RE
    SUFFERING FROM AN
    EXTREME CASE OF…
    ENVIRONMENT
    DISPARITY

    View Slide

  10. INFRASTRUCTURE
    YOUR
    STUFF
    !
    YOUR PROBLEM YOUR PROBLEM
    NOT
    CONTAINER
    IMAGE
    YOUR
    STUFF
    REQUIRED
    ENVIRONMENT

    View Slide

  11. MY
    PROBLEM
    kirill_makes_pics
    https://pixabay.com/photos/despaired-businessman-business-2261021/

    View Slide

  12. As a web project owner, I can freely
    configure system components in
    my environment, so that I can
    operate a wide range of different
    applications with different system
    requirements.

    View Slide

  13. > ID=$(mw app install contao --version=^4.13 -q)
    > mw app dependency update $ID \
    --set 'php=^8.2' \
    --set 'graphicsmagick=*' \
    --set 'nodejs=^20'
    TASK:
    BUILD A CONTAINER IMAGE
    FROM THIS USER-PROVIDED SPECIFICATION
    UNRELATED: https://github.com/mittwald/cli

    View Slide

  14. FROM ubuntu:22.04
    RUN apt-get update -y
    RUN apt-get install –y php
    RUN apt-get install –y graphicsmagick
    RUN apt-get install –y nodejs
    INTRODUCING THE
    CONTAINERFILE

    View Slide

  15. FROM ubuntu:22.04
    RUN apt-get update -y
    {{- range $software, $version := .UserRequirements }}
    RUN apt-get install -y {{ $software }}={{ $version }}
    {{- end }}
    IS THIS A GOOD IDEA?
    (spoiler alert: it is, indeed, not)
    (actually, WAY more complicated)

    View Slide

  16. php=^8.2
    graphicsmagick=*
    nodejs=^20
    FROM ubuntu 22.04
    {{- range $sw, $v := .User
    apt-get install -y {{ $sw
    {{- end }}
    USER ENVIRONMENT
    SPECIFICATION
    TEMPLATED
    CONTAINERFILE
    (actually, WAY more complicated)
    CONTAINER
    IMAGE
    DOCKER BUILD
    HARBOR
    DOCKER
    PUSH
    DOCKER
    PULL
    CONTAINER
    REGISTRY
    USER
    PROFIT
    💰

    View Slide

  17. > ID=$(mw app install contao --version=^4.13 -q)
    > mw app dependency update $ID \
    --set 'php=^8.2' \
    --set 'graphicsmagick=*' \
    --set 'nodejs=^20'
    ⏳ updating environment…
    ✅ environment updated after 27m51s
    🫠

    View Slide

  18. kirill_makes_pics
    https://pixabay.com/photos/despaired-businessman-business-2261021/

    View Slide

  19. COMMON
    CONTAINER BUILD
    OPTIMIZATION
    TARGETS
    📐 Image size
    ⏱ Build Time
    🏎 Download time
    yellowcat
    https://pixabay.com/photos/sport-clock-jogging-time-fitness-3874743/

    View Slide

  20. MEH. WHO CARES? 😐

    View Slide

  21. CONTAINER
    RUNTIME
    PULL / RUN
    BUILD
    ENVIRONMENT
    CONTAINER
    REGISTRY
    PUSH
    BUILD IMAGE SPEC
    https://github.com/opencontainers/image-
    spec
    DISTRIBUTION SPEC
    https://github.com/opencontainers/distribution-
    spec
    RUNTIME SPEC
    https://github.com/opencontainers/runtime-
    spec

    View Slide

  22. View Slide

  23. file1 file2
    file3
    file2 file4
    FILESYSTEM
    LAYERS
    UNION MOUNT
    (OVERLAYFS or AUFS) file1 file2 file3 file4

    View Slide

  24. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json src /var/www/html/
    RUN composer install --no-dev
    # to be continued…
    CONTAINERFILE

    View Slide

  25. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json src /var/www/html/
    RUN composer install --no-dev
    # to be continued…
    CACHED
    UNCACHED
    A change invalidates all
    cached layers after this one
    PRO TIP Pull your base image before
    building to make sure it’s current #security
    PRO TIP Most of the time, you
    don’t need dev dependencies
    in your container image.

    View Slide

  26. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json src /var/www/html/
    RUN composer install --no-dev
    # to be continued… This file changes (presumably)
    relatively frequently
    This file changes
    (presumably)
    relatively infrequently
    This step needs only to run
    when the composer.json
    changed

    View Slide

  27. FROM ubuntu:22.04
    RUN apt-get update –y
    RUN apt-get install –y composer php-fpm
    COPY composer.json /var/www/html/
    RUN composer install --no-dev
    COPY src /var/www/html/
    # to be continued…
    CACHED
    UNCACHED
    Build instruction order
    determines cache utilization!

    View Slide

  28. FROM ubuntu:22.04 AS builder
    RUN apt-get update –y
    RUN apt-get install –y composer
    COPY composer.json app/
    RUN composer install --no-dev
    FROM ubuntu:22.04
    RUN apt-get update –y && apt-get install –y php-fpm
    COPY --from=builder /var/www/html/vendor ↵
    /var/www/html/
    COPY src /var/www/html/
    # to be continued…
    CACHED
    UNCACHED
    Use multistage builds to
    optimize even further
    CACHED

    View Slide

  29. composer.json auth.json
    vendor/
    auth.json
    COPY composer.json auth.json /app
    RUN composer install --no-dev
    RUN rm auth.json
    composer.json auth.json vendor/
    DON’T
    DO THIS!
    Your credentials are still there!

    View Slide

  30. composer.json auth.json
    vendor/
    COPY composer.json auth.json /app
    RUN composer install --no-dev
    vendor/
    DO
    THIS, instead
    And keep your credentials safe!
    FROM AS builder
    FROM
    vendor/
    COPY --from=base vendor vendor

    View Slide

  31. REGULAR DOCKER
    USERS
    ME
    https://knowyourmeme.com/memes/two-guys-on-a-bus

    View Slide

  32. CONFIGURATION (JSON)
    sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744
    FS LAYERS
    sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7
    sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d
    sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b
    sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4
    sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9
    sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f
    sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f
    sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216
    sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8
    sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f
    sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476
    sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d
    sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736
    sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8
    sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9
    (USUAL FILE FORMATS: TAR or TAR.GZ)
    CONTENTS
    OF A
    DOCKER
    IMAGE
    MANIFEST

    View Slide

  33. INDEX
    (JSON)
    amd64
    armv7
    arm64
    MANIFEST
    CONFIGURATION (JSON)
    sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744
    FS LAYERS
    sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7
    sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d
    sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b
    sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4
    sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9
    sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f
    sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f
    sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216
    sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8
    sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f
    sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476
    sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d
    sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736
    sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8
    sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9
    (USUAL FILE FORMATS: TAR or TAR.GZ)

    View Slide

  34. amd64
    armv7
    arm64
    GET /v2/{image}/manifests/{tag}
    Accept: application/vnd.oci.image.index.v1+json
    {
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "schemaVersion": 2,
    "manifests": [
    {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:f10dac435406078cfc491daa0cd99598b10909ae259a4e13d…",
    "size": 3145,
    "platform": {
    "architecture": "arm",
    "os": "linux",
    "variant": "v7"
    }
    },
    […]
    ]
    }
    INDEX
    (JSON)

    View Slide

  35. GET /v2/{image}/manifests/{digest}
    Accept: application/vnd.oci.image.manifest.v1+json
    {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "schemaVersion": 2,
    "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:966da77cd61840d8d91d34b9cbdae2104683f93be5550f9819b…",
    "size": 14426
    },
    "layers": [
    {
    "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
    "digest": "sha256:faef57eae888cbe4a5613eca6741b5e48d768b83f6088858a…",
    "size": 29124829
    },
    […]
    ]
    }
    MANIFEST
    ION
    63d1504addbe7eb6b82e7798fb744
    S
    6b87074e55fea6bd1ecae066693c7
    13d066d60543ac1b176802dd8638d
    6e956b4c0bd9f0c29aaf1b9c5b49b
    9716aa4f8df2d148dc1f947b1abe4
    995ba25193b0f43ba8bbc4abd99f9
    0daf28b405740eb498f9c07837f4f
    007953eb385eb7c8425461fb2a54f
    a3ede53659006c66744b472a95216
    367ede931b8e2a5272c562cdcfee8
    7e60cb9b50efb1ead4d05d35f079f
    343953d068478845c22870c596476
    f5e07dc2fd02f0c1b5931ba2b230d
    960290328e8829987721c61143736
    f9b1e9aee0d51a5320cd5ba6d60a8
    f578a8647c7811807d0a2723916b9
    R or TAR.GZ)

    View Slide

  36. “IT’S JUST A REST API,
    ISN’T IT?”
    r.HandleFunc(
    "/v2/{repo}/{image}/manifests/{tag}",
    func(w http.ResponseWriter, r *http.Response) {
    repo, image, tag := manifestRequestVars(r)
    if manifest, ok := GetManifest(repo, image, tag); ok {
    json.NewEncoder(w).Write(manifest)
    }
    },
    )
    r.HandleFunc(
    "/v2/{repo}/{image}/blobs/{digest}",
    func(w http.ResponseWriter, r *http.Response) {
    repo, image, digest := blobRequestVars(r)
    file, err := OpenBlob(repo, image, digest)
    if err != nil {
    // handle error
    }
    io.Copy(file, r)
    },
    )

    View Slide

  37. php=^8.2
    graphicsmagick=*
    nodejs=^20
    USER ENVIRONMENT
    SPECIFICATION
    PRE-PACKAGED,
    INDEPENDENT (!) LAYERS
    (VIRTUAL)
    CONTAINER
    IMAGE
    GENERATE
    IMAGE MANIFEST
    CUSTOM
    REGISTRY
    SERVE VIA
    CUSTOM API
    DOCKER
    PULL
    CONTAINER
    REGISTRY
    PROFIT
    💰
    USER
    BUILT
    AHEAD-OF-TIME

    View Slide

  38. ThMilherou
    https://pixabay.com/photos/workshop-carpenter-crafts-artisan-4863393/
    TOOL
    RECOMMENDATIONS

    View Slide

  39. DIVE (https://github.com/wagoodman/dive)
    Explore individual layers in a given image

    View Slide

  40. SKOPEO (https://github.com/containers/skopeo)
    Inspect (and manipulate) images in remote registries

    View Slide

  41. SLIM TOOLKIT (https://github.com/slimtoolkit/slim)
    Optimize images for size, using black magic (requires caution!)

    View Slide

  42. HOOKED ON CONTAINERS?
    https://github.com/mittwald/feature-requests/issues/1

    View Slide

  43. END
    THE
    Three-shots
    https://pixabay.com/photos/ocean-blue-whale-sea-wave-whale-2051760/

    View Slide

  44. BONUS
    SLIDES

    View Slide

  45. FROM big-ass-image:v1.2.3 AS source
    FROM scratch
    COPY --from=source / /
    # NOTE: Make sure to set environment variables
    CMD …
    ENTRYPOINT …
    SQUASHING
    IMAGES

    View Slide

  46. SQUASHING
    IMAGES
    ORIGINAL IMAGE: 1Gi
    SQUASHED IMAGE: 400Mi

    View Slide

  47. SQUASHING
    IMAGES
    ORIGINAL IMAGE: 1Gi
    DOWNLOAD ON LAYER CHANGE: 10Mi
    SQUASHED IMAGE: 400Mi
    DOWNLOAD ON LAYER CHANGE: 400Mi

    View Slide

  48. DATA CENTER
    NETWORKING
    STORAGE
    VIRTUALIZATION
    OS
    RUNTIME
    APPLICATION
    DATA CENTER
    NETWORKING
    STORAGE
    VIRTUALIZATION
    OS
    RUNTIME
    APPLICATION
    DATA CENTER
    NETWORKING
    STORAGE
    VIRTUALIZATION
    OS
    RUNTIME
    APPLICATION
    ON PREMISE IaaS PaaS
    YOUR PROBLEM
    NOT YOUR
    PROBLEM
    MORE CONTROL
    MORE PROBLEMS
    LESS CONTROL
    LESS PROBLEMS

    View Slide

  49. DATA CENTER
    NETWORKING
    STORAGE
    VIRTUALIZATION
    OS
    RUNTIME
    APPLICATION
    ON PREMISE
    DATA CENTER
    NETWORKING
    STORAGE
    VIRTUALIZATION
    OS
    RUNTIME
    APPLICATION
    IaaS
    DATA CENTER
    NETWORKING
    STORAGE
    VIRTUALIZATION
    OS
    RUNTIME
    APPLICATION
    PaaS
    YOUR PROBLEM
    NOT YOUR
    PROBLEM
    CaaS
    DATA CENTER
    NETWORKING
    STORAGE
    VIRTUALIZATION
    OS
    CONTAINER RUNTIME
    CONTAINER OS
    RUNTIME
    APPLICATION
    MORE CONTROL
    MORE PROBLEMS
    LESS CONTROL
    LESS PROBLEMS

    View Slide