Slide 1

Slide 1 text

docker-record
 A Semi-Automated Approach from Container Setup to Dockerfile Jürgen Cito Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe

Slide 2

Slide 2 text

Docker Hackathon in New York City 
 WINNER: docker-record 
 @citostyle, @allonhadaya Tool to address shortcoming in setup and maintenance process Open Source on GitHub: https://github.com/citostyle/docker-record

Slide 3

Slide 3 text

Dockerfile Definition of infrastructure and dependencies of a container through instructions # Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"] Dependencies Base Image Install Open Port Start Server

Slide 4

Slide 4 text

The Ideal Container World App Idea A Dockerfile magically appears # Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"] Build Image Run Containers ??? Profit

Slide 5

Slide 5 text

The Real Container World # Adding some config to nginx FROM nginx:1.9.5 We need certificates!! # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, …

Slide 6

Slide 6 text

The Real Container World # Adding some config to nginx FROM nginx:1.9.5 Test fail Server names are too long # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, … # Configure Nginx and apply fix for very long server names RUN echo "daemon off;" >> /etc/nginx/nginx.conf \ && sed -i 's/^http {/&\n server_names_hash_bucket_size 128;/g' /etc/nginx/ nginx.conf

Slide 7

Slide 7 text

The Real Container World # Adding some config to nginx FROM nginx:1.9.5 Test fail Volume for certificates # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, … # Configure Nginx and apply fix for very long server names RUN echo "daemon off;" >> /etc/nginx/nginx.conf \ && sed -i 's/^http {/&\n server_names_hash_bucket_size 128;/g' /etc/nginx/ nginx.conf ... VOLUME ["/etc/nginx/certs"] ...

Slide 8

Slide 8 text

Container Setup & Maintenance Initial Setup Configuration Tuning in Container Run Tests, Trial&Error Helplessly identify what has actually been done within the container and what files have been touched and translate it to Dockerfile instructions. Test the new image, pray it worked out

Slide 9

Slide 9 text

docker-record To The Rescue! Photo Credits: José María Pérez Nuñez, https://flic.kr/p/ksiHGZ

Slide 10

Slide 10 text

Usage docker-record container Container should be running first Starts an interactive (bash) session and ‘records’ what you are doing

Slide 11

Slide 11 text

Usage docker-record container --replay Generates Dockerfile by ‘replaying’ your previously recorded sessions

Slide 12

Slide 12 text

Behind the Scenes Photo Credits: Jonathan Kos-Read, https://flic.kr/p/cepP61

Slide 13

Slide 13 text

Instrumentation For every started session we gather the execution context > User Id > Bash Command > Working Directory > Environment Variables

Slide 14

Slide 14 text

Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')"

Slide 15

Slide 15 text

Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Start bash within bash an pass init ‘file’

Slide 16

Slide 16 text

Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Define function within ‘init file’

Slide 17

Slide 17 text

Record - Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Record context in temporary files

Slide 18

Slide 18 text

Record - Instrumentation docker exec -ti container bash -c "bash --init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Debug mode ‘trap’ catches signal and executes our function

Slide 19

Slide 19 text

Replay - Extraction & Filter > Not all bash commands lead to Dockerfile statement > Blacklisting, Whitelisting > Heuristics for mapping execution context

Slide 20

Slide 20 text

Replay - Mapping Heuristics (1/3) CMD instruction - there is only one in a Dockerfile > Defines an exectuable for the start of a container > Whitelisting paths that are likely to contain an executable

Slide 21

Slide 21 text

Replay - Mapping Heuristics (2/3) Match Editors that have been used - indicates a file that has been changed > Whitelist for possible editors > Check whether opened files have been actually edited in container > Copy file from container and add an ADD instruction

Slide 22

Slide 22 text

Replay - Mapping Heuristics (3/3) Everything else that hasn’t been filtered gets executed > RUN instruction for every remaining extracted command

Slide 23

Slide 23 text

docker-record in Action Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015 docker run -ti --name=demo -p 80:80 ubuntu /bin/bash 1. Run container (image: Ubuntu, container name: demo)

Slide 24

Slide 24 text

docker-record in Action Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015 docker-record demo
 root@2208baba1afc:/# 2. Start interactive shell with docker-record (Every command you issue now will be instrumented)

Slide 25

Slide 25 text

docker-record in Action Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015 https://www.digitalocean.com/community/tutorials/docker-explained-how-to-containerize-and-use-nginx-as-a-proxy 3. Follow the tutorial to install and configure nginx

Slide 26

Slide 26 text

docker-record in Action Walking through a tutorial by DigitalOcean - Screencast: vimeo.com/139944015 docker-record demo --replay 4. Exit container and generate Dockerfile RUN apt-get update RUN apt-get install -y nano wget dialog net-tools RUN apt-get upgrade -y nginx RUN rm -v /etc/nginx/nginx.conf ADD ./build/_etc_nginx_nginx.conf /etc/nginx/nginx.conf CMD service nginx start

Slide 27

Slide 27 text

Future Work Heuristics Domain Knowledge
 > Better filter to suggest build context
 > Suggesting VOLUMES Dependency Analysis > Reordering/grouping of instructions

Slide 28

Slide 28 text

Future Work
 Refactoring, Refactoring, Refactoring… # make it work now, refactor later

Slide 29

Slide 29 text

Supported by the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 610802 (CloudWave). www.cloudwave-fp7.eu

Slide 30

Slide 30 text

docker-record
 A Semi-Automated Approach from Container Setup to Dockerfile Jürgen Cito Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe @citostyle Slides: speakerdeck.com/citostyle https://github.com/citostyle/docker-record