docker-record

`docker-record` is the local Docker NYC Hackathon (#DockerHackDay3) winner in September 2015. The slides have been presented at the Docker Meetup in Zurich, Switzerland.

Github: https://www.github.com/citostyle/docker-record

Transcript

1. docker-record
 A Semi-Automated Approach from Container Setup to Dockerfile Jürgen Cito Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe
2. Docker Hackathon in New York City 
 WINNER: docker-record 
 @citostyle, @allonhadaya Tool to address shortcoming in setup and maintenance process Open Source on GitHub: https://github.com/citostyle/docker-record
3. Dockerfile Definition of infrastructure and dependencies of a container through instructions # Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"] Dependencies Base Image Install Open Port Start Server
4. The Ideal Container World App Idea A Dockerfile magically appears # Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"] Build Image Run Containers ??? Profit
5. The Real Container World # Adding some config to nginx FROM nginx:1.9.5 We need certificates!! # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, …
6. The Real Container World # Adding some config to nginx FROM nginx:1.9.5 Test fail Server names are too long # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, … # Configure Nginx and apply fix for very long server names RUN echo "daemon off;" >> /etc/nginx/nginx.conf \ && sed -i 's/^http {/&\n server_names_hash_bucket_size 128;/g' /etc/nginx/ nginx.conf
7. The Real Container World # Adding some config to nginx FROM nginx:1.9.5 Test fail Volume for certificates # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, … # Configure Nginx and apply fix for very long server names RUN echo "daemon off;" >> /etc/nginx/nginx.conf \ && sed -i 's/^http {/&\n server_names_hash_bucket_size 128;/g' /etc/nginx/ nginx.conf ... VOLUME ["/etc/nginx/certs"] ...
8. Container Setup & Maintenance Initial Setup Configuration Tuning in Container Run Tests, Trial&Error Helplessly identify what has actually been done within the container and what files have been touched and translate it to Dockerfile instructions. Test the new image, pray it worked out
9. docker-record To The Rescue! Photo Credits: José María Pérez Nuñez, https://flic.kr/p/ksiHGZ
10. Usage docker-record container Container should be running first Starts an interactive (bash) session and ‘records’ what you are doing
11. Usage docker-record container --replay Generates Dockerfile by ‘replaying’ your previously recorded sessions
12. Behind the Scenes Photo Credits: Jonathan Kos-Read, https://flic.kr/p/cepP61
13. Instrumentation For every started session we gather the execution context > User Id > Bash Command > Working Directory > Environment Variables
14. Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')"
15. Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Start bash within bash an pass init ‘file’
16. Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Define function within ‘init file’
17. Record - Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Record context in temporary files
18. Record - Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Debug mode ‘trap’ catches signal and executes our function
19. Replay - Extraction & Filter > Not all bash commands lead to Dockerfile statement > Blacklisting, Whitelisting > Heuristics for mapping execution context
20. Replay - Mapping Heuristics (1/3) CMD instruction - there is only one in a Dockerfile > Defines an exectuable for the start of a container > Whitelisting paths that are likely to contain an executable
21. Replay - Mapping Heuristics (2/3) Match Editors that have been used - indicates a file that has been changed > Whitelist for possible editors > Check whether opened files have been actually edited in container > Copy file from container and add an ADD instruction
22. Replay - Mapping Heuristics (3/3) Everything else that hasn’t been filtered gets executed > RUN instruction for every remaining extracted command
23. docker-record in Action Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015 docker run -ti --name=demo -p 80:80 ubuntu /bin/bash 1. Run container (image: Ubuntu, container name: demo)
24. docker-record in Action Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015 docker-record demo
 root@2208baba1afc:/# 2. Start interactive shell with docker-record (Every command you issue now will be instrumented)
25. docker-record in Action Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015 https://www.digitalocean.com/community/tutorials/docker-explained-how-to-containerize-and-use-nginx-as-a-proxy 3. Follow the tutorial to install and configure nginx
26. docker-record in Action Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015 docker-record demo --replay 4. Exit container and generate Dockerfile RUN apt-get update RUN apt-get install -y nano wget dialog net-tools RUN apt-get upgrade -y nginx RUN rm -v /etc/nginx/nginx.conf ADD ./build/_etc_nginx_nginx.conf /etc/nginx/nginx.conf CMD service nginx start
27. Future Work Heuristics Domain Knowledge
 > Better filter to suggest build context
 > Suggesting VOLUMES Dependency Analysis > Reordering/grouping of instructions
28. Future Work
 Refactoring, Refactoring, Refactoring… # make it work now, refactor later
29. Supported by the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 610802 (CloudWave). www.cloudwave-fp7.eu
30. docker-record
 A Semi-Automated Approach from Container Setup to Dockerfile Jürgen Cito <cito@ifi.uzh.ch> Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe @citostyle Slides: speakerdeck.com/citostyle https://github.com/citostyle/docker-record