Using Docker Containers to Improve Reproducibility in Software Engineering Research

Transcript

1. ICSE’16 Technical Briefing, May 17th, Austin, TX Photo Credits: Astrid

   Westvang, https://flic.kr/p/pWJLCW Using Docker Containers to Improve Reproducibility in Software Engineering Research Jürgen Cito, Harald C. Gall

2. Jürgen Cito PhD @ UZH Harald Gall Prof @ UZH

   Software Evolution Cloud-based Software Engineering Human Factors in Software Engineering

3. This technical briefing… Conceptual, more abstract notion of Reproducibility in

   Research Concrete instructions to aid Reproducibility in Research

4. Reproducibility is the ability of an entire experiment or study

   to be duplicated, either by the same researcher or by someone else working independently. Reproducing an experiment is called replicating it. What is Reproducibility? No research paper can ever be considered to be the final word, and the replication and corroboration of research results is key to the scientific process.

5. What is Reproducibility? Repeatability of a certain process in order

   to establish a fact or the conditions under which we are able to observe the same fact* A process to share methods, describe the environment, 
 in order to recreate results. * Mockus et al. “Experiences from replicating a case study to investigate reproducibility of software development.”

6. Scientific Process Experiment Design Data Collection Data Analysis Interpret Results

   Hypotheses Reproducibility

7. What is Reproducibility? Establishing facts
 > Steps (or method) to

   establish the fact
 > Sharing computational knowledge Controlling environment
 > Execution Environment
 > Dependencies Providing data
 > Ability to interpret data
 > Computational Analyses Low barriers to replicate
 > Comprehensible results
 > Ease of achieving replication

8. What is Reproducibility in SE Research? Algorithms / Computational Analyses

   Developed Tools or Prototypes Quantitative Evaluations + internal knowledge of the necessary process to derive/establish results


9. Artifact Evaluation & Replication Packages In Software Engineering Research:
 -

   FSE’15/16, MSR’15/16 In Programming Languages Research:
 - PLDI, POPL, OOPSLA

10. Current State of Sharing Artifacts in SE Researcher’s website

11. Case Study: ChangeDistiller https://bitbucket.org/sealuzh/tools-changedistiller Research Project

12. Why is reproducibility hard? Why does it fail?

13. Reasons for Failed Reproducibility (1/2) Here is a link to

    download the code for this paper. Good luck trying to download the only post-doc who knows how to run this thing* * Paraphrase of a tweet I cannot seem to find anymore

14. Source: Collberg et al., “Measuring Reproducibility in Computer Systems Research”,

    http://reproducibility.cs.arizona.edu/tr.pdf Reasons for Failed Reproducibility (2/2)

15. Case Study: ChangeDistiller https://bitbucket.org/sealuzh/tools-changedistiller Research Project - Developed 2006-2009
 How

    many Java versions have we passed?
 - Dependencies defined in a Maven pom file
 Are they all still available in the repository?
 - How does analysis in ChangeDistiller work?
 What is the entry point?

16. Challenges in Reproducibility > No standard way of describing experiments,

    environments, (derived) data, and workflows > No transparency in creating environments and the steps/methods to establish facts or recreate analysis 
 > Experimental nature of research code and ecosystems makes it often hard to build > Unresolved or undocumented dependencies > Infrastructure for storage and distribution

17. … to the rescue

18. Docker Containers to the rescue (1/2) What is Docker? Docker

    allows you to package an application with all of its dependencies into a standardized unit for software development* Containers consist of everything that enables software to run:
 > Code
 > Runtime
 > System Tools
 > System Libraries * https://www.docker.com/what-docker

19. Docker Containers to the rescue (2/2) What can Docker be

    for SE research? Docker allows you to package a 
 - Prototype
 - Proof-of-concept Implementation
 - Computational analysis or experiment 
 with all of its dependencies into a standardized unit for reproducible research * https://www.docker.com/what-docker

20. https://www.docker.com/what-is-docker Technical Overview / Virtual Machines vs Containers “Lightweight” VM


    > Container is an isolated process 
 (“chroot on steroids”)
 > Own process space
 > Own network interface
 > Feels like a VM
 > Share kernel with the host
 > Isolation through cgroups/namespaces https://blog.docker.com/2016/03/containers-are-not-vms/

21. Docker Engine Centralized runtime environment for containers Enables portability Sole

    dependency for Docker No Emulation layer (almost no performance impact) https://www.docker.com/products/docker-engine

22. Benefits of Docker Containers Fast instantiation (~1-3 seconds)
 Almost native

    performance
 Transparent build process
 Smaller Images
 Easy to build, share, and publish * https://www.docker.com/what-docker also compared to
 other container technology

23. Local Docker Workflow # Build redis from source # Make

    sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http:// dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build- essential libc6-dev tcl wget RUN wget http://download.redis.io/redis- stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis- server"] CMD ["--dir", "/redis-data"] Dockerfile build Image Docker Image Docker Container run

24. Terminology Dockerfile
 Declarative definition of an environment for producing an

    image
 
 Docker Image
 Immutable artifact built from a Dockerfile, has one to many layers. Docker Container
 Execution environment - Instantiation/running version of an image (can be parameterized) Docker Registry
 Public or private repository that stores allows for distribution of images
 (Docker Hub - https://hub.docker.com/ or CoreOS Quay - https://quay.io/)
 


25. Local Docker Workflow # Build redis from source # Make

    sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http:// dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build- essential libc6-dev tcl wget RUN wget http://download.redis.io/redis- stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis- server"] CMD ["--dir", "/redis-data"] Dockerfile build Image Docker Image Docker Container run

26. Dockerfile Definition of infrastructure and dependencies of a container through

    instructions # Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) COPY redis.conf /var/www/redis.conf RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"] Dependencies Base Image Install Open Port Start Server Volume Base Image can be an OS (Ubuntu) or a different, existing image Runs commands as if you were typing them in the command line Copies local files from build context into container

27. Data Volumes A specially-designated directory within one or more containers

    that bypasses the Union File System*
 
 Volumes allow you to manage data within containers
 > Mount a host directory (dependency to the host filesystem)
 > Mount a data volume container (dependency to another container)
 > Mount a shared-storage volume (NFS, iSCSI, etc.) * https://docs.docker.com/engine/userguide/containers/dockervolumes/

28. Local Docker Workflow # Build redis from source # Make

    sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http:// dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build- essential libc6-dev tcl wget RUN wget http://download.redis.io/redis- stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis- server"] CMD ["--dir", "/redis-data"] Dockerfile build Image Docker Image Docker Container run

29. Dockerfile —> Image Definition of infrastructure and dependencies of a

    container through instructions docker build -t <imagename> . Build Context containing all local dependencies and Dockerfile

30. Docker Images # docker images REPOSITORY TAG IMAGE ID CREATED

    SIZE mhart/alpine-node latest 2a15d8568f75 1 week ago 36.76 MB hakyll latest d575da1e730c 2 weeks ago 1.487 GB redis alpine 50405530a7e5 4 weeks ago 15.95 MB Lists all previously built images # docker rmi hakyll Untagged: hakyll:latest Deleted: sha256:3240943c9ea3f72db51… Deleted: sha256:a3aeefae0d4b8f61… Deleted: sha256:16a7ebd378002f1261… Removes image ‘hakyll’ and all its layers from disk

31. Local Docker Workflow # Build redis from source # Make

    sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http:// dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build- essential libc6-dev tcl wget RUN wget http://download.redis.io/redis- stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis- server"] CMD ["--dir", "/redis-data"] Dockerfile build Image Docker Image Docker Container run

32. Image —> Container docker run -d --name <containername> -p 80:5000

    <imagename> Run container in the background
 (d for daemon) https://docs.docker.com/engine/reference/run/ Give the container a unique name Port mapping
 First the exposed port (80) Second the port within the container (5000) Many more possibilities to run containers, see full reference here: (A typical example)

33. Container Management docker ps List all running containers docker ps

    -a List all containers (also stopped) docker stop <container> Stop a running container docker rm <container> Remove a stopped container

34. Container Debugging # docker run -ti --entrypoint=bash <imagename> Start image

    with a different entrypoint # docker exec -ti <container> bash Start an interactive shell into a running container # docker inspect <container> Low-level information on a container or image

35. Docker Hub: Public Registry

36. Pulling Docker Images
 Getting started with existing images docker pull

    nginx:latest

37. Pulling Docker Images
 Getting started with existing images docker pull

    nginx:latest Reference to a Docker Image in the Docker Hub

38. Pulling Docker Images
 Getting started with existing images docker pull

    nginx:latest Images can have many “tags”

39. Pulling Docker Images
 Getting started with existing images docker pull

    nginx:latest Pulls an image from a Docker registry

40. Pushing Docker Images to a Registry
 Tag Image docker tag

    c6fdd6639541 <username>/<imagename>:<tagname> Image Id 
 (retrieve through ) docker images

41. Pushing Docker Images to a Registry
 Push Image docker login

    --username=<username> --email=<email> docker push <username>/<imagename>:<tagname>

42. Case Study: ChangeDistiller https://bitbucket.org/sealuzh/tools-changedistiller Research Project # Build redis from

    source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile FROM ubuntu:12.04 MAINTAINER dockerfiles http:// dockerfiles.github.io RUN echo "deb http://archive.ubuntu.com/ ubuntu precise main universe" > /etc/apt/ sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y gcc make g++ build- essential libc6-dev tcl wget RUN wget http://download.redis.io/redis- stable.tar.gz -O - | tar -xvz # RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test) RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379 ENTRYPOINT ["/redis-stable/src/redis- server"] CMD ["--dir", "/redis-data"] Dockerfile build Image Docker Image Docker Container run

43. Recap: Challenges in Reproducibility > No standard way of describing

    experiments, environments, (derived) data, and workflows > No transparency in creating environments and the steps/methods to establish facts or recreate analysis
 
 > Experimental nature of research code and ecosystems makes it often hard to build > Unresolved or undocumented dependencies > Infrastructure for storage and distribution Dockerfile Docker Image Docker Container Registries (Docker Hub, Quay, …) Dockerfile

44. Limitations > Performance sensitivity 
 [Jimenez et al., The Role

    of Container Technology in Reproducible Computer Systems Research] > Proprietary Software and Dependencies > Non-Disclosure Agreements / Intellectual Property
 
 > Can we build the same artifact from the specification (Dockerfile) even in 10 years? [Suggestion: Version Pinning]

45. Conclusions > Containers enable a standard, fast, and easy way

    of describing experiments and environments > Helps your future self, reviewers, and other researchers to make use of your work

46. Using Docker Containers to Improve Reproducibility in SE Research Jürgen

    Cito <cito@ifi.uzh.ch>, Harald Gall <gall@ifi.uzh.ch> Photo Credits: Nan Palmero, https://flic.kr/p/nPLSpe @citostyle Slides: speakerdeck.com/citostyle Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW