The Dockerfile explosion - and the need for higher level tools

Slide 1

Slide 1 text

The Dockerfile explosion Gareth Rushgrove Senior Software Engineer Puppet

Slide 2

Slide 2 text

The Dockerfile explosion and the need for higher level tools

Slide 3

Slide 3 text

Introductions Who am I and what am I doing here

Slide 4

Slide 4 text

@garethr

Slide 5

Slide 5 text

(without introducing more risk) Gareth Rushgrove

Slide 6

Slide 6 text

Built the Puppet Docker module

Slide 7

Slide 7 text

Maintain the Puppet images

Slide 8

Slide 8 text

Obsessed with metadata

Slide 9

Slide 9 text

A brief history of Dockerfile

Slide 10

Slide 10 text

Docker can build images automatically by reading the instructions from a Dockerfile From the ofﬁcial docs at https://docs.docker.com/engine/reference/builder/

Slide 11

Slide 11 text

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. From the ofﬁcial docs at https://docs.docker.com/engine/reference/builder/

Slide 12

Slide 12 text

A simple Dockerfile FROM ubuntu # Install vnc, xvfb in order to create a 'fake' display and fire RUN apt-get update && apt-get install -y x11vnc xvfb firefox RUN mkdir ~/.vnc # Setup a password RUN x11vnc -storepasswd 1234 ~/.vnc/passwd # Autostart firefox (might not be the best way, but it does the RUN bash -c 'echo "firefox" >> /.bashrc' EXPOSE 5900 CMD ["x11vnc", "-forever", "-usepw", "-create"]

Slide 13

Slide 13 text

Dockerfile reference

Slide 14

Slide 14 text

Commands you know MAINTAINER RUN CMD ["executable","param1","param2"] EXPOSE [...] ADD ... ENV WORKDIR /path/to/workdir USER daemon VOLUME ["/data"] ENTRYPOINT ["executable", "param1", “param2"] COPY ...

Slide 15

Slide 15 text

Commands you don’t know ONBUILD [INSTRUCTION] STOPSIGNAL signal ARG [=] LABEL = = = … HEALTHCHECK [OPTIONS] CMD command SHELL ["executable", "parameters"]

Slide 16

Slide 16 text

Close ALL the issues

Slide 17

Slide 17 text

Although this is not a definitive move, we temporarily won’t accept more patches to the Dockerfile syntax Docker Inc

Slide 18

Slide 18 text

HEALTHCHECK coming in 1.12

Slide 19

Slide 19 text

SHELL coming in 1.12

Slide 20

Slide 20 text

Why Dockerfiles are great

Slide 21

Slide 21 text

Simplicity FROM scratch COPY hello / CMD ["/hello"]

Slide 22

Slide 22 text

Multi-platform support PS> Install-PackageProvider ContainerImage -Force PS> Install-ContainerImage -Name WindowsServerCore PS> docker images REPOSITORY TAG IMAGE ID CREA windowsservercore 10.0.14300.1000 dbfee88ee9fd 7 we

Slide 23

Slide 23 text

Linting

Slide 24

Slide 24 text

Editor support

Slide 25

Slide 25 text

Why Dockerfiles are problematic

Slide 26

Slide 26 text

Complexity RUN apt-get update && \ apt-get install -y wget=1.17.1-1ubuntu1 && \ wget https://apt.example.com/release-"$UBUNTU_CODENAME".deb dpkg -i release-"$UBUNTU_CODENAME".deb && \ rm release-"$UBUNTU_CODENAME".deb && \ apt-get update && \ apt-get install --no-install-recommends -y package=0.1.2 && apt-get clean && \ rm -rf /var/lib/apt/lists/*

Slide 27

Slide 27 text

Dockerfile proliferation

Slide 28

Slide 28 text

language:Dockerfile maintainer

Slide 29

Slide 29 text

138,062

Slide 30

Slide 30 text

Only two approaches to reuse

Slide 31

Slide 31 text

Inheritance FROM debian:jessie

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Dockerfile is not the source of truth for your image

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

The Dockerfile generally works beautifully for the class of problem for which it was designed Nathan Leclair, Docker Inc

Slide 36

Slide 36 text

Nathan Leclair, Docker Inc The Dockerfile is a tool for creating images, but it is not the only weapon in your arsenal

Slide 37

Slide 37 text

Putting the problems in context

Slide 38

Slide 38 text

If we dockerize all of our applications how many Dockerfiles is that?

Slide 39

Slide 39 text

If we build a complex hierarchy of Dockerfiles, how quickly can we trace/rebuild a specific image?

Slide 40

Slide 40 text

As best-practices develops how can we refactor our Dockefiles with confidence?

Slide 41

Slide 41 text

Are Dockerfiles best managed centrally or on a team-by-team basis?

Slide 42

Slide 42 text

Some community ideas

Slide 43

Slide 43 text

Generate Dockerfiles

Slide 44

Slide 44 text

Build Dockerfiles with OCAML

Slide 45

Slide 45 text

let base = let email = "[email protected]" in comment "Generated by OCaml Dockerfile" @@ from "ubuntu" ~tag:"trusty" @@ maintainer "Anil Madhavapeddy <%s>" email let ocaml_ubuntu_image = base @@ run "apt-get -y -qq update" @@ run "apt-get -y install ocaml ocaml-native-compilers camlp4-ext onbuild (run "apt-get -y -qq update") ;; OCAML example

Slide 46

Slide 46 text

With Gradle

Slide 47

Slide 47 text

Or Javascript

Slide 48

Slide 48 text

Or Scala and SBT

Slide 49

Slide 49 text

Or with Python

Slide 50

Slide 50 text

- Powerful abstractions - Mature language tooling PROS - Need to compile down to Dockerfile - Everyone has their favourite language CONS

Slide 51

Slide 51 text

No Dockerfile to be seen

Slide 52

Slide 52 text

Docker Image Specification

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Packer

Slide 55

Slide 55 text

{ "builders":[{ "type": "docker", "image": "ubuntu", "export_path": "image.tar" }], "provisioners":[ { "type": "shell", "inline": ["apt-get -y update; apt-get install -y puppet-co }, { Packer example

Slide 56

Slide 56 text

Source-to-Image

Slide 57

Slide 57 text

$ s2i create $ s2i build [] [flags] $ s2i rebuild [] $ s2i usage [flags] $ s2i build ./sinatra-app openshift/ruby-20-centos7 ruby-app s2i example

Slide 58

Slide 58 text

Nix

Slide 59

Slide 59 text

dockerTools.buildImage { name = "redis"; runAsRoot = '' #!${stdenv.shell} ${dockerTools.shadowSetup} groupadd -r redis useradd -r -g redis -d /data -M redis mkdir /data chown redis:redis /data ''; contents = [ redis ]; Nix example

Slide 60

Slide 60 text

Habitat

Slide 61

Slide 61 text

- Powerful PROS - OCI image spec not final - Higher barrier to entry than Dockerfile - Limited support for things like labels CONS

Slide 62

Slide 62 text

Expand on Dockerfile

Slide 63

Slide 63 text

Rocker

Slide 64

Slide 64 text

Rocker adds some crucial features that are missing from Dockerfile while keeping Docker’s original design

Slide 65

Slide 65 text

FROM ubuntu:16.04 MAINTAINER Gareth Rushgrove "[email protected]" ENV PUPPET_AGENT_VERSION="1.5.0" UBUNTU_CODENAME="xenial" PATH=/ LABEL com.puppet.version="0.1.0" com.puppet.dockerfile="/Dockerf MOUNT /opt/puppetlabs /etc/puppetlabs /root/.gem RUN apt-get update && \ apt-get install -y wget=1.17.1-1ubuntu1 && \ wget https://apt.puppetlabs.com/puppetlabs-release-pc1-"$UBU Rockerfile example

Slide 66

Slide 66 text

Slide 67

Slide 67 text

rm -rf /var/lib/apt/lists/* EXPOSE 80 CMD ["nginx"] COPY Rockerfile /Dockerfile TAG puppet/puppet-rocker-example More new instructions

Slide 68

Slide 68 text

Dockramp

Slide 69

Slide 69 text

Dockerfile pre-processors

Slide 70

Slide 70 text

$ cat Dockerfile FROM ubuntu:16.04 MAINTAINER Gareth Rushgrove "[email protected]" ENV PUPPET_AGENT_VERSION="1.5.0" R10K_VERSION="2.2.2" \ UBUNTU_C PUPPET_INSTALL PUPPET_COPY_PUPPETFILE PUPPET_COPY_MANIFESTS PUPPET_RUN EXPOSE 80 Domain-specific extensions

Slide 71

Slide 71 text

$ cat Dockerfile | dockerfilepp FROM ubuntu:16.04 MAINTAINER Gareth Rushgrove "[email protected]" ENV PUPPET_AGENT_VERSION="1.5.0" R10K_VERSION="2.2.2" UBUNTU_COD RUN apt-get update && \ apt-get install -y wget=1.17.1-1ubuntu1 && \ wget https://apt.puppetlabs.com/puppetlabs-release-pc1-"$UBU dpkg -i puppetlabs-release-pc1-"$UBUNTU_CODENAME".deb && \ rm puppetlabs-release-pc1-"$UBUNTU_CODENAME".deb && \ apt-get update && \ Simple expansion

Slide 72

Slide 72 text

- Simple and familiar - Great proving ground for upstream PROS - Still line-oriented - Limited tooling available (yet) CONS