Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ContaoConference23: The Anatomy Of A Docker Image

ContaoConference23: The Anatomy Of A Docker Image

Over the past few years, container technologies (like Docker) have become essential to many a developer's workflow -- be it for local development, testing, CI or production rollouts. In this talk, I will shed light on one of the underlying principles of Docker: the OCI image and distribution specifications. These describe the file format of a Docker image and the API of a Docker registry. Understanding these will help you understand what's actually going on when you run commands like "docker build" or "docker pull". I will also present a case study of a project in which we used these specifications to build a container image solution that implemented features that would have been impossible to implement using the regular tools.

Martin Helmich

October 24, 2023
Tweet

More Decks by Martin Helmich

Other Decks in Technology

Transcript

  1. MARTIN HELMICH Head of Architecture & Chief Tech Evangelist Lecturer,

    Software Engineering & Cloud Computing Sci-Fi-Nerd, Metalhead, Amateur Woodworker
  2. PROD PREPROD ACCEPTANCE INTEGRATION TEST ONE APP MANY ENVIRONMENTS DEVELOPER

    ALICE DEVELOPER BOB DEVELOPER CHARLIE YOUR STUFF
  3. As a web project owner, I can freely configure system

    components in my environment, so that I can operate a wide range of different applications with different system requirements.
  4. > ID=$(mw app install contao --version=^4.13 -q) > mw app

    dependency update $ID \ --set 'php=^8.2' \ --set 'graphicsmagick=*' \ --set 'nodejs=^20' TASK: BUILD A CONTAINER IMAGE FROM THIS USER-PROVIDED SPECIFICATION UNRELATED: https://github.com/mittwald/cli
  5. FROM ubuntu:22.04 RUN apt-get update -y RUN apt-get install –y

    php RUN apt-get install –y graphicsmagick RUN apt-get install –y nodejs INTRODUCING THE CONTAINERFILE
  6. FROM ubuntu:22.04 RUN apt-get update -y {{- range $software, $version

    := .UserRequirements }} RUN apt-get install -y {{ $software }}={{ $version }} {{- end }} IS THIS A GOOD IDEA? (spoiler alert: it is, indeed, not) (actually, WAY more complicated)
  7. php=^8.2 graphicsmagick=* nodejs=^20 FROM ubuntu 22.04 {{- range $sw, $v

    := .User apt-get install -y {{ $sw {{- end }} USER ENVIRONMENT SPECIFICATION TEMPLATED CONTAINERFILE (actually, WAY more complicated) CONTAINER IMAGE DOCKER BUILD HARBOR DOCKER PUSH DOCKER PULL CONTAINER REGISTRY USER PROFIT 💰
  8. > ID=$(mw app install contao --version=^4.13 -q) > mw app

    dependency update $ID \ --set 'php=^8.2' \ --set 'graphicsmagick=*' \ --set 'nodejs=^20' ⏳ updating environment… ✅ environment updated after 27m51s 🫠
  9. COMMON CONTAINER BUILD OPTIMIZATION TARGETS 📐 Image size ⏱ Build

    Time 🏎 Download time yellowcat https://pixabay.com/photos/sport-clock-jogging-time-fitness-3874743/
  10. CONTAINER RUNTIME PULL / RUN BUILD ENVIRONMENT CONTAINER REGISTRY PUSH

    BUILD IMAGE SPEC https://github.com/opencontainers/image- spec DISTRIBUTION SPEC https://github.com/opencontainers/distribution- spec RUNTIME SPEC https://github.com/opencontainers/runtime- spec
  11. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y

    composer php-fpm COPY composer.json src /var/www/html/ RUN composer install --no-dev # to be continued… CONTAINERFILE
  12. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y

    composer php-fpm COPY composer.json src /var/www/html/ RUN composer install --no-dev # to be continued… CACHED UNCACHED A change invalidates all cached layers after this one PRO TIP Pull your base image before building to make sure it’s current #security PRO TIP Most of the time, you don’t need dev dependencies in your container image.
  13. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y

    composer php-fpm COPY composer.json src /var/www/html/ RUN composer install --no-dev # to be continued… This file changes (presumably) relatively frequently This file changes (presumably) relatively infrequently This step needs only to run when the composer.json changed
  14. FROM ubuntu:22.04 RUN apt-get update –y RUN apt-get install –y

    composer php-fpm COPY composer.json /var/www/html/ RUN composer install --no-dev COPY src /var/www/html/ # to be continued… CACHED UNCACHED Build instruction order determines cache utilization!
  15. FROM ubuntu:22.04 AS builder RUN apt-get update –y RUN apt-get

    install –y composer COPY composer.json app/ RUN composer install --no-dev FROM ubuntu:22.04 RUN apt-get update –y && apt-get install –y php-fpm COPY --from=builder /var/www/html/vendor ↵ /var/www/html/ COPY src /var/www/html/ # to be continued… CACHED UNCACHED Use multistage builds to optimize even further CACHED
  16. composer.json auth.json vendor/ auth.json COPY composer.json auth.json /app RUN composer

    install --no-dev RUN rm auth.json composer.json auth.json vendor/ DON’T DO THIS! Your credentials are still there!
  17. composer.json auth.json vendor/ COPY composer.json auth.json /app RUN composer install

    --no-dev vendor/ DO THIS, instead And keep your credentials safe! FROM <base> AS builder FROM <base> vendor/ COPY --from=base vendor vendor
  18. CONFIGURATION (JSON) sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744 FS LAYERS sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7 sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4 sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9

    sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216 sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8 sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476 sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736 sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8 sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9 (USUAL FILE FORMATS: TAR or TAR.GZ) CONTENTS OF A DOCKER IMAGE MANIFEST
  19. INDEX (JSON) amd64 armv7 arm64 MANIFEST CONFIGURATION (JSON) sha256:de8c1b639eb844b3797fb09b55284c85eec63d1504addbe7eb6b82e7798fb744 FS

    LAYERS sha256:3ae0c06b4d3aa97d7e0829233dd36cea1666b87074e55fea6bd1ecae066693c7 sha256:d37d52eeb040d2d490dd824c341b25153cd13d066d60543ac1b176802dd8638d sha256:1004c1ea98aac76b5196cb9cdb0a3b837f66e956b4c0bd9f0c29aaf1b9c5b49b sha256:00817655af3bc75137fae25bebdc1e51e749716aa4f8df2d148dc1f947b1abe4 sha256:d2eae06ed188cc03844f62831094d533bce995ba25193b0f43ba8bbc4abd99f9 sha256:32465e990ab2ee84430d8d588624a04b6270daf28b405740eb498f9c07837f4f sha256:ad8068a097d006952782488d2f168f20ed9007953eb385eb7c8425461fb2a54f sha256:d9ca373df594426b85d6e7b530d1ca4a9eca3ede53659006c66744b472a95216 sha256:f0ba23a8da8db9d5a3e36221c0be2596415367ede931b8e2a5272c562cdcfee8 sha256:7d5f87e10d1d0da3994320de550c20fddf17e60cb9b50efb1ead4d05d35f079f sha256:f9886766cf5acdb5f17b90ddcccc8c1ef7b343953d068478845c22870c596476 sha256:499c8e530c63b627f40a82e046ca2f5e186f5e07dc2fd02f0c1b5931ba2b230d sha256:b943d856aeec1d1949b3afae20476413dc1960290328e8829987721c61143736 sha256:a542bccf4abb5813719832d1c97959d24c0f9b1e9aee0d51a5320cd5ba6d60a8 sha256:3efc822907b02ca22ffaed54c042afd8d0df578a8647c7811807d0a2723916b9 (USUAL FILE FORMATS: TAR or TAR.GZ)
  20. amd64 armv7 arm64 GET /v2/{image}/manifests/{tag} Accept: application/vnd.oci.image.index.v1+json { "mediaType": "application/vnd.oci.image.index.v1+json",

    "schemaVersion": 2, "manifests": [ { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:f10dac435406078cfc491daa0cd99598b10909ae259a4e13d…", "size": 3145, "platform": { "architecture": "arm", "os": "linux", "variant": "v7" } }, […] ] } INDEX (JSON)
  21. GET /v2/{image}/manifests/{digest} Accept: application/vnd.oci.image.manifest.v1+json { "mediaType": "application/vnd.oci.image.manifest.v1+json", "schemaVersion": 2, "config":

    { "mediaType": "application/vnd.oci.image.config.v1+json", "digest": "sha256:966da77cd61840d8d91d34b9cbdae2104683f93be5550f9819b…", "size": 14426 }, "layers": [ { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": "sha256:faef57eae888cbe4a5613eca6741b5e48d768b83f6088858a…", "size": 29124829 }, […] ] } MANIFEST ION 63d1504addbe7eb6b82e7798fb744 S 6b87074e55fea6bd1ecae066693c7 13d066d60543ac1b176802dd8638d 6e956b4c0bd9f0c29aaf1b9c5b49b 9716aa4f8df2d148dc1f947b1abe4 995ba25193b0f43ba8bbc4abd99f9 0daf28b405740eb498f9c07837f4f 007953eb385eb7c8425461fb2a54f a3ede53659006c66744b472a95216 367ede931b8e2a5272c562cdcfee8 7e60cb9b50efb1ead4d05d35f079f 343953d068478845c22870c596476 f5e07dc2fd02f0c1b5931ba2b230d 960290328e8829987721c61143736 f9b1e9aee0d51a5320cd5ba6d60a8 f578a8647c7811807d0a2723916b9 R or TAR.GZ)
  22. “IT’S JUST A REST API, ISN’T IT?” r.HandleFunc( "/v2/{repo}/{image}/manifests/{tag}", func(w

    http.ResponseWriter, r *http.Response) { repo, image, tag := manifestRequestVars(r) if manifest, ok := GetManifest(repo, image, tag); ok { json.NewEncoder(w).Write(manifest) } }, ) r.HandleFunc( "/v2/{repo}/{image}/blobs/{digest}", func(w http.ResponseWriter, r *http.Response) { repo, image, digest := blobRequestVars(r) file, err := OpenBlob(repo, image, digest) if err != nil { // handle error } io.Copy(file, r) }, )
  23. php=^8.2 graphicsmagick=* nodejs=^20 USER ENVIRONMENT SPECIFICATION PRE-PACKAGED, INDEPENDENT (!) LAYERS

    (VIRTUAL) CONTAINER IMAGE GENERATE IMAGE MANIFEST CUSTOM REGISTRY SERVE VIA CUSTOM API DOCKER PULL CONTAINER REGISTRY PROFIT 💰 USER BUILT AHEAD-OF-TIME
  24. FROM big-ass-image:v1.2.3 AS source FROM scratch COPY --from=source / /

    # NOTE: Make sure to set environment variables CMD … ENTRYPOINT … SQUASHING IMAGES
  25. SQUASHING IMAGES ORIGINAL IMAGE: 1Gi DOWNLOAD ON LAYER CHANGE: 10Mi

    SQUASHED IMAGE: 400Mi DOWNLOAD ON LAYER CHANGE: 400Mi
  26. DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION DATA CENTER

    NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION ON PREMISE IaaS PaaS YOUR PROBLEM NOT YOUR PROBLEM MORE CONTROL MORE PROBLEMS LESS CONTROL LESS PROBLEMS
  27. DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION ON PREMISE

    DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION IaaS DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS RUNTIME APPLICATION PaaS YOUR PROBLEM NOT YOUR PROBLEM CaaS DATA CENTER NETWORKING STORAGE VIRTUALIZATION OS CONTAINER RUNTIME CONTAINER OS RUNTIME APPLICATION MORE CONTROL MORE PROBLEMS LESS CONTROL LESS PROBLEMS