Upgrade to Pro — share decks privately, control downloads, hide ads and more …

docker-record

Jürgen Cito
November 30, 2015

 docker-record

`docker-record` is the local Docker NYC Hackathon (#DockerHackDay3) winner in September 2015. The slides have been presented at the Docker Meetup in Zurich, Switzerland.

Github: https://www.github.com/citostyle/docker-record

Jürgen Cito

November 30, 2015
Tweet

More Decks by Jürgen Cito

Other Decks in Programming

Transcript

  1. docker-record

    A Semi-Automated Approach from Container Setup to Dockerfile
    Jürgen Cito
    Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe

    View full-size slide

  2. Docker Hackathon in New York City

    WINNER: docker-record 

    @citostyle, @allonhadaya
    Tool to address shortcoming in setup and
    maintenance process
    Open Source on GitHub:
    https://github.com/citostyle/docker-record

    View full-size slide

  3. Dockerfile
    Definition of infrastructure and dependencies of a container through instructions
    # Build redis from source
    # Make sure you have the redis source code checked out in
    # the same directory as this Dockerfile
    FROM ubuntu:12.04
    MAINTAINER dockerfiles http://dockerfiles.github.io
    RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/
    sources.list
    RUN apt-get update
    RUN apt-get upgrade -y
    RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
    RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
    # RUN tar -zvzf /redis/redis-stable.tar.gz
    RUN (cd /redis-stable && make)
    RUN (cd /redis-stable && make test)
    RUN mkdir -p /redis-data
    VOLUME ["/redis-data"]
    EXPOSE 6379
    ENTRYPOINT ["/redis-stable/src/redis-server"]
    CMD ["--dir", "/redis-data"]
    Dependencies
    Base Image
    Install
    Open Port
    Start Server

    View full-size slide

  4. The Ideal Container World
    App Idea
    A Dockerfile
    magically appears
    # Build redis from source
    # Make sure you have the redis source code checked out in
    # the same directory as this Dockerfile
    FROM ubuntu:12.04
    MAINTAINER dockerfiles http://dockerfiles.github.io
    RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/
    sources.list
    RUN apt-get update
    RUN apt-get upgrade -y
    RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget
    RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz
    # RUN tar -zvzf /redis/redis-stable.tar.gz
    RUN (cd /redis-stable && make)
    RUN (cd /redis-stable && make test)
    RUN mkdir -p /redis-data
    VOLUME ["/redis-data"]
    EXPOSE 6379
    ENTRYPOINT ["/redis-stable/src/redis-server"]
    CMD ["--dir", "/redis-data"]
    Build Image Run Containers
    ???
    Profit

    View full-size slide

  5. The Real Container World
    # Adding some config to nginx
    FROM nginx:1.9.5 We need
    certificates!!
    # Install wget and install/updates certificates
    RUN apt-get update \
    && apt-get install -y -q --no-install-recommends \
    ca-certificates \
    wget \
    && apt-get clean \
    Build new image
    Run new container
    Run Tests,
    Trial&Error,

    View full-size slide

  6. The Real Container World
    # Adding some config to nginx
    FROM nginx:1.9.5
    Test fail
    Server names
    are too long
    # Install wget and install/updates certificates
    RUN apt-get update \
    && apt-get install -y -q --no-install-recommends \
    ca-certificates \
    wget \
    && apt-get clean \
    Build new image
    Run new container
    Run Tests,
    Trial&Error,

    # Configure Nginx and apply fix for very long server
    names
    RUN echo "daemon off;" >> /etc/nginx/nginx.conf \
    && sed -i 's/^http {/&\n
    server_names_hash_bucket_size 128;/g' /etc/nginx/
    nginx.conf

    View full-size slide

  7. The Real Container World
    # Adding some config to nginx
    FROM nginx:1.9.5
    Test fail
    Volume for
    certificates
    # Install wget and install/updates certificates
    RUN apt-get update \
    && apt-get install -y -q --no-install-recommends \
    ca-certificates \
    wget \
    && apt-get clean \
    Build new image
    Run new container
    Run Tests,
    Trial&Error,

    # Configure Nginx and apply fix for very long server
    names
    RUN echo "daemon off;" >> /etc/nginx/nginx.conf \
    && sed -i 's/^http {/&\n
    server_names_hash_bucket_size 128;/g' /etc/nginx/
    nginx.conf
    ...
    VOLUME ["/etc/nginx/certs"]
    ...

    View full-size slide

  8. Container Setup & Maintenance
    Initial
    Setup
    Configuration
    Tuning in
    Container
    Run Tests,
    Trial&Error
    Helplessly identify what
    has actually been done
    within the container and
    what files have been
    touched and translate it
    to Dockerfile
    instructions. Test the
    new image, pray it
    worked out

    View full-size slide

  9. docker-record To The Rescue!
    Photo Credits: José María Pérez Nuñez, https://flic.kr/p/ksiHGZ

    View full-size slide

  10. Usage
    docker-record container
    Container should
    be running first
    Starts an interactive (bash) session
    and ‘records’ what you are doing

    View full-size slide

  11. Usage
    docker-record container --replay
    Generates Dockerfile by ‘replaying’
    your previously recorded sessions

    View full-size slide

  12. Behind the Scenes
    Photo Credits: Jonathan Kos-Read, https://flic.kr/p/cepP61

    View full-size slide

  13. Instrumentation
    For every started session we gather the execution context
    > User Id
    > Bash Command
    > Working Directory
    > Environment Variables

    View full-size slide

  14. Instrumentation
    docker exec -ti container bash -c "bash --init-file <(echo
    ‘SESSION_ROOT=/tmp/record;
    mkdir -p $SESSION_ROOT;
    function __instrument() {
    echo $EUID >> $SESSION_ROOT/euid &&
    echo $BASH_COMMAND >> $SESSION_ROOT/cmd &&
    echo $PWD >> $SESSION_ROOT/pwd &&
    echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env ||
    exit;
    };
    shopt -s extdebug;
    trap __instrument DEBUG;'')"

    View full-size slide

  15. Instrumentation
    docker exec -ti container bash -c "bash --init-file <(echo
    ‘SESSION_ROOT=/tmp/record;
    mkdir -p $SESSION_ROOT;
    function __instrument() {
    echo $EUID >> $SESSION_ROOT/euid &&
    echo $BASH_COMMAND >> $SESSION_ROOT/cmd &&
    echo $PWD >> $SESSION_ROOT/pwd &&
    echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env ||
    exit;
    };
    shopt -s extdebug;
    trap __instrument DEBUG;'')"
    Start bash within bash an pass init ‘file’

    View full-size slide

  16. Instrumentation
    docker exec -ti container bash -c "bash --init-file <(echo
    ‘SESSION_ROOT=/tmp/record;
    mkdir -p $SESSION_ROOT;
    function __instrument() {
    echo $EUID >> $SESSION_ROOT/euid &&
    echo $BASH_COMMAND >> $SESSION_ROOT/cmd &&
    echo $PWD >> $SESSION_ROOT/pwd &&
    echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env ||
    exit;
    };
    shopt -s extdebug;
    trap __instrument DEBUG;'')"
    Define function within
    ‘init file’

    View full-size slide

  17. Record - Instrumentation
    docker exec -ti container bash -c "bash --init-file <(echo
    ‘SESSION_ROOT=/tmp/record;
    mkdir -p $SESSION_ROOT;
    function __instrument() {
    echo $EUID >> $SESSION_ROOT/euid &&
    echo $BASH_COMMAND >> $SESSION_ROOT/cmd &&
    echo $PWD >> $SESSION_ROOT/pwd &&
    echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env ||
    exit;
    };
    shopt -s extdebug;
    trap __instrument DEBUG;'')"
    Record context in
    temporary files

    View full-size slide

  18. Record - Instrumentation
    docker exec -ti container bash -c "bash --init-file <(echo
    ‘SESSION_ROOT=/tmp/record;
    mkdir -p $SESSION_ROOT;
    function __instrument() {
    echo $EUID >> $SESSION_ROOT/euid &&
    echo $BASH_COMMAND >> $SESSION_ROOT/cmd &&
    echo $PWD >> $SESSION_ROOT/pwd &&
    echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env ||
    exit;
    };
    shopt -s extdebug;
    trap __instrument DEBUG;'')"
    Debug mode
    ‘trap’ catches signal and
    executes our function

    View full-size slide

  19. Replay - Extraction & Filter
    > Not all bash commands lead to Dockerfile statement
    > Blacklisting, Whitelisting
    > Heuristics for mapping execution context

    View full-size slide

  20. Replay - Mapping Heuristics
    (1/3)
    CMD instruction - there is only one in a Dockerfile
    > Defines an exectuable for the start of a container
    > Whitelisting paths that are likely to contain an executable

    View full-size slide

  21. Replay - Mapping Heuristics
    (2/3)
    Match Editors that have been used - indicates a file that has
    been changed
    > Whitelist for possible editors
    > Check whether opened files have been actually edited in container
    > Copy file from container and add an ADD instruction

    View full-size slide

  22. Replay - Mapping Heuristics
    (3/3)
    Everything else that hasn’t been filtered gets executed
    > RUN instruction for every remaining extracted command

    View full-size slide

  23. docker-record in Action
    Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015
    docker run -ti --name=demo -p 80:80 ubuntu /bin/bash
    1. Run container
    (image: Ubuntu, container name: demo)

    View full-size slide

  24. docker-record in Action
    Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015
    docker-record demo

    root@2208baba1afc:/#
    2. Start interactive shell with docker-record
    (Every command you issue now will be instrumented)

    View full-size slide

  25. docker-record in Action
    Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015
    https://www.digitalocean.com/community/tutorials/docker-explained-how-to-containerize-and-use-nginx-as-a-proxy
    3. Follow the tutorial to install and configure nginx

    View full-size slide

  26. docker-record in Action
    Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015
    docker-record demo --replay
    4. Exit container and generate Dockerfile
    RUN apt-get update
    RUN apt-get install -y nano wget dialog net-tools
    RUN apt-get upgrade -y nginx
    RUN rm -v /etc/nginx/nginx.conf
    ADD ./build/_etc_nginx_nginx.conf /etc/nginx/nginx.conf
    CMD service nginx start

    View full-size slide

  27. Future Work
    Heuristics
    Domain Knowledge

    > Better filter to suggest build context

    > Suggesting VOLUMES
    Dependency Analysis
    > Reordering/grouping of instructions

    View full-size slide

  28. Future Work

    Refactoring, Refactoring, Refactoring…
    # make it work now, refactor later

    View full-size slide

  29. Supported by the European Community’s
    Seventh Framework Programme (FP7/2007-2013)
    under grant agreement no. 610802 (CloudWave).
    www.cloudwave-fp7.eu

    View full-size slide

  30. docker-record

    A Semi-Automated Approach from Container Setup to Dockerfile
    Jürgen Cito
    Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe
    @citostyle
    Slides:
    speakerdeck.com/citostyle
    https://github.com/citostyle/docker-record

    View full-size slide