Share
Embed
docker-record
`docker-record` is the local Docker NYC Hackathon (#DockerHackDay3) winner in September 2015. The slides have been presented at the Docker Meetup in Zurich, Switzerland.
Github: https://www.github.com/citostyle/docker-record
Jürgen Cito
November 30, 2015
Transcript
30 slides
-
docker-record A Semi-Automated Approach from Container Setup to Dockerfile Jürgen
Cito Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe -
Docker Hackathon in New York City WINNER: docker-record
@citostyle, @allonhadaya Tool to address shortcoming in setup and maintenance process Open Source on GitHub: https://github.com/citostyle/docker-record -
Dockerfile Definition of infrastructure and dependencies of a container through
instructions # Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"] Dependencies Base Image Install Open Port Start Server -
The Ideal Container World App Idea A Dockerfile magically appears
# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"] Build Image Run Containers ??? Profit -
The Real Container World # Adding some config to nginx
FROM nginx:1.9.5 We need certificates!! # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, … -
The Real Container World # Adding some config to nginx
FROM nginx:1.9.5 Test fail Server names are too long # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, … # Configure Nginx and apply fix for very long server names RUN echo "daemon off;" >> /etc/nginx/nginx.conf \ && sed -i 's/^http {/&\n server_names_hash_bucket_size 128;/g' /etc/nginx/ nginx.conf -
The Real Container World # Adding some config to nginx
FROM nginx:1.9.5 Test fail Volume for certificates # Install wget and install/updates certificates RUN apt-get update \ && apt-get install -y -q --no-install-recommends \ ca-certificates \ wget \ && apt-get clean \ Build new image Run new container Run Tests, Trial&Error, … # Configure Nginx and apply fix for very long server names RUN echo "daemon off;" >> /etc/nginx/nginx.conf \ && sed -i 's/^http {/&\n server_names_hash_bucket_size 128;/g' /etc/nginx/ nginx.conf ... VOLUME ["/etc/nginx/certs"] ... -
Container Setup & Maintenance Initial Setup Configuration Tuning in Container
Run Tests, Trial&Error Helplessly identify what has actually been done within the container and what files have been touched and translate it to Dockerfile instructions. Test the new image, pray it worked out -
docker-record To The Rescue! Photo Credits: José María Pérez Nuñez,
https://flic.kr/p/ksiHGZ -
Usage docker-record container Container should be running first Starts an
interactive (bash) session and ‘records’ what you are doing -
Instrumentation For every started session we gather the execution context
> User Id > Bash Command > Working Directory > Environment Variables -
Instrumentation docker exec -ti container bash -c "bash --init-file <(echo
‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" -
Instrumentation docker exec -ti container bash -c "bash --init-file <(echo
‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Start bash within bash an pass init ‘file’ -
Instrumentation docker exec -ti container bash -c "bash --init-file <(echo
‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Define function within ‘init file’ -
Record - Instrumentation docker exec -ti container bash -c "bash
--init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Record context in temporary files -
Record - Instrumentation docker exec -ti container bash -c "bash
--init-file <(echo ‘SESSION_ROOT=/tmp/record; mkdir -p $SESSION_ROOT; function __instrument() { echo $EUID >> $SESSION_ROOT/euid && echo $BASH_COMMAND >> $SESSION_ROOT/cmd && echo $PWD >> $SESSION_ROOT/pwd && echo $(export | tr "\\n" ";") >> $SESSION_ROOT/env || exit; }; shopt -s extdebug; trap __instrument DEBUG;'')" Debug mode ‘trap’ catches signal and executes our function -
Replay - Extraction & Filter > Not all bash commands
lead to Dockerfile statement > Blacklisting, Whitelisting > Heuristics for mapping execution context -
Replay - Mapping Heuristics (1/3) CMD instruction - there is
only one in a Dockerfile > Defines an exectuable for the start of a container > Whitelisting paths that are likely to contain an executable -
Replay - Mapping Heuristics (2/3) Match Editors that have been
used - indicates a file that has been changed > Whitelist for possible editors > Check whether opened files have been actually edited in container > Copy file from container and add an ADD instruction -
Replay - Mapping Heuristics (3/3) Everything else that hasn’t been
filtered gets executed > RUN instruction for every remaining extracted command -
docker-record in Action Walking through a tutorial by DigitalOcean -
Screencast: vimeo.com/139944015 docker run -ti --name=demo -p 80:80 ubuntu /bin/bash 1. Run container (image: Ubuntu, container name: demo) -
docker-record in Action Walking through a tutorial by DigitalOcean -
Screencast: vimeo.com/139944015 docker-record demo root@2208baba1afc:/# 2. Start interactive shell with docker-record (Every command you issue now will be instrumented) -
docker-record in Action Walking through a tutorial by DigitalOcean -
Screencast: vimeo.com/139944015 https://www.digitalocean.com/community/tutorials/docker-explained-how-to-containerize-and-use-nginx-as-a-proxy 3. Follow the tutorial to install and configure nginx -
docker-record in Action Walking through a tutorial by DigitalOcean -
Screencast: vimeo.com/139944015 docker-record demo --replay 4. Exit container and generate Dockerfile RUN apt-get update RUN apt-get install -y nano wget dialog net-tools RUN apt-get upgrade -y nginx RUN rm -v /etc/nginx/nginx.conf ADD ./build/_etc_nginx_nginx.conf /etc/nginx/nginx.conf CMD service nginx start -
Future Work Heuristics Domain Knowledge > Better filter to suggest
build context > Suggesting VOLUMES Dependency Analysis > Reordering/grouping of instructions -
Supported by the European Community’s Seventh Framework Programme (FP7/2007-2013) under
grant agreement no. 610802 (CloudWave). www.cloudwave-fp7.eu -
docker-record A Semi-Automated Approach from Container Setup to Dockerfile Jürgen
Cito <cito@ifi.uzh.ch> Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe @citostyle Slides: speakerdeck.com/citostyle https://github.com/citostyle/docker-record