$30 off During Our Annual Pro Sale. View Details »

docker-record

Jürgen Cito
November 30, 2015

 docker-record

`docker-record` is the local Docker NYC Hackathon (#DockerHackDay3) winner in September 2015. The slides have been presented at the Docker Meetup in Zurich, Switzerland.

Github: https://www.github.com/citostyle/docker-record

Jürgen Cito

November 30, 2015
Tweet

More Decks by Jürgen Cito

Other Decks in Programming

Transcript

  1. docker-record
 A Semi-Automated Approach from Container Setup to Dockerfile Jürgen

    Cito Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe
  2. Docker Hackathon in New York City 
 WINNER: docker-record 


    @citostyle, @allonhadaya Tool to address shortcoming in setup and maintenance process Open Source on GitHub: https://github.com/citostyle/docker-record
  3. Dockerfile Definition of infrastructure and dependencies of a container through

    instructions # Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"] Dependencies Base Image Install Open Port Start Server
  4. The Ideal Container World App Idea A Dockerfile magically appears

    # Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"] Build Image Run Containers ??? Profit
  5. The Real Container World # Adding some config to nginx

    FROM nginx:1.9.5 We need certificates!! # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, …
  6. The Real Container World # Adding some config to nginx

    FROM nginx:1.9.5 Test fail Server names are too long # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, … # Configure Nginx and apply fix for very long server names RUN echo "daemon off;" >> /etc/nginx/nginx.conf \ && sed -i 's/^http {/&\n server_names_hash_bucket_size 128;/g' /etc/nginx/ nginx.conf
  7. The Real Container World # Adding some config to nginx

    FROM nginx:1.9.5 Test fail Volume for certificates # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, … # Configure Nginx and apply fix for very long server names RUN echo "daemon off;" >> /etc/nginx/nginx.conf \ && sed -i 's/^http {/&\n server_names_hash_bucket_size 128;/g' /etc/nginx/ nginx.conf ... VOLUME ["/etc/nginx/certs"] ...
  8. Container Setup & Maintenance Initial Setup Configuration Tuning in Container

    Run Tests, Trial&Error Helplessly identify what has actually been done within the container and what files have been touched and translate it to Dockerfile instructions. Test the new image, pray it worked out
  9. Usage docker-record container Container should be running first Starts an

    interactive (bash) session and ‘records’ what you are doing
  10. Instrumentation For every started session we gather the execution context

    > User Id > Bash Command > Working Directory > Environment Variables
  11. Instrumentation docker exec -ti container bash -c "bash --init-file <(echo

    ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')"
  12. Instrumentation docker exec -ti container bash -c "bash --init-file <(echo

    ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Start bash within bash an pass init ‘file’
  13. Instrumentation docker exec -ti container bash -c "bash --init-file <(echo

    ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Define function within ‘init file’
  14. Record - Instrumentation docker exec -ti container bash -c "bash

    --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Record context in temporary files
  15. Record - Instrumentation docker exec -ti container bash -c "bash

    --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Debug mode ‘trap’ catches signal and executes our function
  16. Replay - Extraction & Filter > Not all bash commands

    lead to Dockerfile statement > Blacklisting, Whitelisting > Heuristics for mapping execution context
  17. Replay - Mapping Heuristics (1/3) CMD instruction - there is

    only one in a Dockerfile > Defines an exectuable for the start of a container > Whitelisting paths that are likely to contain an executable
  18. Replay - Mapping Heuristics (2/3) Match Editors that have been

    used - indicates a file that has been changed > Whitelist for possible editors > Check whether opened files have been actually edited in container > Copy file from container and add an ADD instruction
  19. Replay - Mapping Heuristics (3/3) Everything else that hasn’t been

    filtered gets executed > RUN instruction for every remaining extracted command
  20. docker-record in Action Walking through a tutorial by DigitalOcean -

    Screencast: vimeo.com/139944015 docker run -ti --name=demo -p 80:80 ubuntu /bin/bash 1. Run container (image: Ubuntu, container name: demo)
  21. docker-record in Action Walking through a tutorial by DigitalOcean -

    Screencast: vimeo.com/139944015 docker-record demo
 root@2208baba1afc:/# 2. Start interactive shell with docker-record (Every command you issue now will be instrumented)
  22. docker-record in Action Walking through a tutorial by DigitalOcean -

    Screencast: vimeo.com/139944015 https://www.digitalocean.com/community/tutorials/docker-explained-how-to-containerize-and-use-nginx-as-a-proxy 3. Follow the tutorial to install and configure nginx
  23. docker-record in Action Walking through a tutorial by DigitalOcean -

    Screencast: vimeo.com/139944015 docker-record demo --replay 4. Exit container and generate Dockerfile RUN apt-get update RUN apt-get install -y nano wget dialog net-tools RUN apt-get upgrade -y nginx RUN rm -v /etc/nginx/nginx.conf ADD ./build/_etc_nginx_nginx.conf /etc/nginx/nginx.conf CMD service nginx start
  24. Future Work Heuristics Domain Knowledge
 > Better filter to suggest

    build context
 > Suggesting VOLUMES Dependency Analysis > Reordering/grouping of instructions
  25. Supported by the European Community’s Seventh Framework Programme (FP7/2007-2013) under

    grant agreement no. 610802 (CloudWave). www.cloudwave-fp7.eu
  26. docker-record
 A Semi-Automated Approach from Container Setup to Dockerfile Jürgen

    Cito <cito@ifi.uzh.ch> Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe @citostyle Slides: speakerdeck.com/citostyle https://github.com/citostyle/docker-record