$30 off During Our Annual Pro Sale. View Details »

The Dockerfile explosion - and the need for higher level tools

The Dockerfile explosion - and the need for higher level tools

Talk from DockerCon 2016. All about the challenges of building Docker images. Discussion of the problems, what's great and not-so-great about Dockerfile, and examples of alternative tooling.

Gareth Rushgrove

June 21, 2016
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. The Dockerfile explosion
    Gareth
    Rushgrove
    Senior Software Engineer
    Puppet

    View Slide

  2. The Dockerfile
    explosion and the
    need for higher
    level tools

    View Slide

  3. Introductions
    Who am I and what am I doing here

    View Slide

  4. @garethr

    View Slide

  5. (without introducing more risk)
    Gareth Rushgrove

    View Slide

  6. Built the Puppet Docker module

    View Slide

  7. Maintain the Puppet images

    View Slide

  8. Obsessed with metadata

    View Slide

  9. A brief history of
    Dockerfile

    View Slide

  10. Docker can build images
    automatically by reading the
    instructions from a Dockerfile
    From the official docs at https://docs.docker.com/engine/reference/builder/

    View Slide

  11. A Dockerfile is a text document
    that contains all the commands a
    user could call on the command
    line to assemble an image.
    From the official docs at https://docs.docker.com/engine/reference/builder/

    View Slide

  12. A simple Dockerfile
    FROM ubuntu
    # Install vnc, xvfb in order to create a 'fake' display and fire
    RUN apt-get update && apt-get install -y x11vnc xvfb firefox
    RUN mkdir ~/.vnc
    # Setup a password
    RUN x11vnc -storepasswd 1234 ~/.vnc/passwd
    # Autostart firefox (might not be the best way, but it does the
    RUN bash -c 'echo "firefox" >> /.bashrc'
    EXPOSE 5900
    CMD ["x11vnc", "-forever", "-usepw", "-create"]

    View Slide

  13. Dockerfile reference

    View Slide

  14. Commands you know
    MAINTAINER
    RUN
    CMD ["executable","param1","param2"]
    EXPOSE [...]
    ADD ...
    ENV
    WORKDIR /path/to/workdir
    USER daemon
    VOLUME ["/data"]
    ENTRYPOINT ["executable", "param1", “param2"]
    COPY ...

    View Slide

  15. Commands you don’t know
    ONBUILD [INSTRUCTION]
    STOPSIGNAL signal
    ARG [=]
    LABEL = = = …
    HEALTHCHECK [OPTIONS] CMD command
    SHELL ["executable", "parameters"]

    View Slide

  16. Close ALL the issues

    View Slide

  17. Although this is not a definitive
    move, we temporarily won’t
    accept more patches to the
    Dockerfile syntax
    Docker Inc

    View Slide

  18. HEALTHCHECK coming in 1.12

    View Slide

  19. SHELL coming in 1.12

    View Slide

  20. Why Dockerfiles
    are great

    View Slide

  21. Simplicity
    FROM scratch
    COPY hello /
    CMD ["/hello"]

    View Slide

  22. Multi-platform support
    PS> Install-PackageProvider ContainerImage -Force
    PS> Install-ContainerImage -Name WindowsServerCore
    PS> docker images
    REPOSITORY TAG IMAGE ID CREA
    windowsservercore 10.0.14300.1000 dbfee88ee9fd 7 we

    View Slide

  23. Linting

    View Slide

  24. Editor support

    View Slide

  25. Why Dockerfiles
    are problematic

    View Slide

  26. Complexity
    RUN apt-get update && \
    apt-get install -y wget=1.17.1-1ubuntu1 && \
    wget https://apt.example.com/release-"$UBUNTU_CODENAME".deb
    dpkg -i release-"$UBUNTU_CODENAME".deb && \
    rm release-"$UBUNTU_CODENAME".deb && \
    apt-get update && \
    apt-get install --no-install-recommends -y package=0.1.2 &&
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

    View Slide

  27. Dockerfile proliferation

    View Slide

  28. language:Dockerfile maintainer

    View Slide

  29. 138,062

    View Slide

  30. Only two approaches to reuse

    View Slide

  31. Inheritance
    FROM debian:jessie

    View Slide

  32. View Slide

  33. Dockerfile is not the source of
    truth for your image

    View Slide

  34. View Slide

  35. The Dockerfile generally works
    beautifully for the class of
    problem for which it was designed
    Nathan Leclair, Docker Inc

    View Slide

  36. Nathan Leclair, Docker Inc
    The Dockerfile is a tool for
    creating images, but it is not the
    only weapon in your arsenal

    View Slide

  37. Putting the
    problems in
    context

    View Slide

  38. If we dockerize all of our
    applications how many
    Dockerfiles is that?

    View Slide

  39. If we build a complex hierarchy of
    Dockerfiles, how quickly can we
    trace/rebuild a specific image?

    View Slide

  40. As best-practices develops how
    can we refactor our Dockefiles
    with confidence?

    View Slide

  41. Are Dockerfiles best
    managed centrally or on a
    team-by-team basis?

    View Slide

  42. Some community
    ideas

    View Slide

  43. Generate
    Dockerfiles

    View Slide

  44. Build Dockerfiles with OCAML

    View Slide

  45. let base =
    let email = "[email protected]" in
    comment "Generated by OCaml Dockerfile" @@
    from "ubuntu" ~tag:"trusty" @@
    maintainer "Anil Madhavapeddy <%s>" email
    let ocaml_ubuntu_image =
    base @@
    run "apt-get -y -qq update" @@
    run "apt-get -y install ocaml ocaml-native-compilers camlp4-ext
    onbuild (run "apt-get -y -qq update") ;;
    OCAML example

    View Slide

  46. With Gradle

    View Slide

  47. Or Javascript

    View Slide

  48. Or Scala and SBT

    View Slide

  49. Or with Python

    View Slide

  50. - Powerful abstractions
    - Mature language tooling
    PROS
    - Need to compile down to Dockerfile
    - Everyone has their favourite language
    CONS

    View Slide

  51. No Dockerfile
    to be seen

    View Slide

  52. Docker Image Specification

    View Slide

  53. View Slide

  54. Packer

    View Slide

  55. {
    "builders":[{
    "type": "docker",
    "image": "ubuntu",
    "export_path": "image.tar"
    }],
    "provisioners":[
    {
    "type": "shell",
    "inline": ["apt-get -y update; apt-get install -y puppet-co
    },
    {
    Packer example

    View Slide

  56. Source-to-Image

    View Slide

  57. $ s2i create
    $ s2i build [] [flags]
    $ s2i rebuild []
    $ s2i usage [flags]
    $ s2i build ./sinatra-app openshift/ruby-20-centos7 ruby-app
    s2i example

    View Slide

  58. Nix

    View Slide

  59. dockerTools.buildImage {
    name = "redis";
    runAsRoot = ''
    #!${stdenv.shell}
    ${dockerTools.shadowSetup}
    groupadd -r redis
    useradd -r -g redis -d /data -M redis
    mkdir /data
    chown redis:redis /data
    '';
    contents = [ redis ];
    Nix example

    View Slide

  60. Habitat

    View Slide

  61. - Powerful
    PROS
    - OCI image spec not final
    - Higher barrier to entry than Dockerfile
    - Limited support for things like labels
    CONS

    View Slide

  62. Expand on
    Dockerfile

    View Slide

  63. Rocker

    View Slide

  64. Rocker adds some crucial
    features that are missing from
    Dockerfile while keeping
    Docker’s original design

    View Slide

  65. FROM ubuntu:16.04
    MAINTAINER Gareth Rushgrove "[email protected]"
    ENV PUPPET_AGENT_VERSION="1.5.0" UBUNTU_CODENAME="xenial" PATH=/
    LABEL com.puppet.version="0.1.0" com.puppet.dockerfile="/Dockerf
    MOUNT /opt/puppetlabs /etc/puppetlabs /root/.gem
    RUN apt-get update && \
    apt-get install -y wget=1.17.1-1ubuntu1 && \
    wget https://apt.puppetlabs.com/puppetlabs-release-pc1-"$UBU
    Rockerfile example

    View Slide

  66. FROM ubuntu:16.04
    MAINTAINER Gareth Rushgrove "[email protected]"
    ENV PUPPET_AGENT_VERSION="1.5.0" UBUNTU_CODENAME="xenial" PATH=/
    LABEL com.puppet.version="0.1.0" com.puppet.dockerfile="/Dockerf
    MOUNT /opt/puppetlabs /etc/puppetlabs /root/.gem
    RUN apt-get update && \
    apt-get install -y wget=1.17.1-1ubuntu1 && \
    wget https://apt.puppetlabs.com/puppetlabs-release-pc1-"$UBU
    Includes new instructions

    View Slide

  67. rm -rf /var/lib/apt/lists/*
    EXPOSE 80
    CMD ["nginx"]
    COPY Rockerfile /Dockerfile
    TAG puppet/puppet-rocker-example
    More new instructions

    View Slide

  68. Dockramp

    View Slide

  69. Dockerfile pre-processors

    View Slide

  70. $ cat Dockerfile
    FROM ubuntu:16.04
    MAINTAINER Gareth Rushgrove "[email protected]"
    ENV PUPPET_AGENT_VERSION="1.5.0" R10K_VERSION="2.2.2" \ UBUNTU_C
    PUPPET_INSTALL
    PUPPET_COPY_PUPPETFILE
    PUPPET_COPY_MANIFESTS
    PUPPET_RUN
    EXPOSE 80
    Domain-specific extensions

    View Slide

  71. $ cat Dockerfile | dockerfilepp
    FROM ubuntu:16.04
    MAINTAINER Gareth Rushgrove "[email protected]"
    ENV PUPPET_AGENT_VERSION="1.5.0" R10K_VERSION="2.2.2" UBUNTU_COD
    RUN apt-get update && \
    apt-get install -y wget=1.17.1-1ubuntu1 && \
    wget https://apt.puppetlabs.com/puppetlabs-release-pc1-"$UBU
    dpkg -i puppetlabs-release-pc1-"$UBUNTU_CODENAME".deb && \
    rm puppetlabs-release-pc1-"$UBUNTU_CODENAME".deb && \
    apt-get update && \
    Simple expansion

    View Slide

  72. - Simple and familiar
    - Great proving ground for upstream
    PROS
    - Still line-oriented
    - Limited tooling available (yet)
    CONS

    View Slide

  73. The future
    Speculation and things I’d like to see

    View Slide

  74. Formal specification for
    Dockerfile

    View Slide

  75. RUN, FROM, COPY, etc.
    as first class API primitives

    View Slide

  76. Opinionated workflow
    tooling around image build

    View Slide

  77. Shared libraries and
    support for pre-processors

    View Slide

  78. Complementary tools that
    take an organization-wide
    view of image building

    View Slide

  79. Conclusions
    If all you take away is…

    View Slide

  80. Dockerfile is a great starting
    point for many use cases

    View Slide

  81. But we will need better tools for
    managing many Dockerfiles

    View Slide

  82. And Dockerfile is just one
    interface to building images

    View Slide

  83. Ultimately we’ll need different types
    of tools for different use cases

    View Slide

  84. Questions?
    And thanks for listening

    View Slide

  85. Thank you!

    View Slide