Docker & radio astronomy - containing fragile scientific software

As a scientific software engineer in the field of radio astronomy, I'm involved in creating, improving and maintaining a broad range of tools used by scientists. These tools are used to process and analyse data coming from various radio telescopes, particularly LOFAR (LOw Frequency ARray) and SKA (Square Kilometre Array). Radio astronomy has a long and rich computing related history, and with that comes a big pile of poorly written, fragile, and badly maintained legacy code. This problem is not unique to radio astronomy but is a common in various scientific fields.

Although no quick solution exists when it comes to re-writing lots of legacy code, Docker helps to contain this fragile software, simplifies the installation and ensures that the software works. In this talk I will discuss how Papino, our umbrella project for containing various radio astronomy libraries and RODRIGUES, an online radio telescope simulator leverage Docker .

Transcript

1. Docker & radio astronomy 1
2. Who Am I Gijs Molenaar Scientific Software Engineer in NL PhD student in SA MSc Artificial Intelligence 2 @gijzelaerr http://pythonic.nl
3. Agenda Radio astronomy Problems in science What is Docker How we use it 3
10. Scientist Q wants to crunch Data X with software Y on Platform Z 10
11. Scientist Q Smart person Self educated on the field of software engineer Gets the job done Good at duct tape programming 11
12. Data X Really big data Datasets of peta, exa bytes LOFAR - 50 GB/s SKA - 10 times global Internet traffic 12
13. Software Y 13 Complex math heavy software Written during PhD of scientist Q Often bad coding practices (Unit) tests? Version control? ‘Runs on my computer’
14. What is docker? ‘Containing’ software Managing containers Connecting containers Deploying containers 14
15. Like virtual machines But not no emulation Process isolation & virtual networking More like chroot/jail 15
17. Image vs container Image set of ‘layers’ read-only stateless build with Dockerfile or commit container container instantiation of image stateful
18. Layered filesytem (unionFS) 18
19. Dockerfile FROM ubuntu:14.04 MAINTAINER gijs@pythonic.nl ENV DEBIAN_FRONTEND noninteractive # enable universe, multiverse, restricted with world wide mirrors ADD apt.sources.list /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y software-properties-common ## add radio astro PPA (with source packages) RUN add-apt-repository -ys ppa:radio-astro/main RUN apt-get update https://github.com/radio-astro/docker-images 19
20. Dockerfile 2 FROM radioastro/base ADD debian_packages / ADD ska_packages / ADD python_packages / # update deb cache RUN apt-get update # install standard debian packages RUN cat /debian_packages | xargs apt-get install -y # installed the packaged software from the PPA RUN cat /ska_packages | xargs apt-get install -y # install all python modules RUN pip install astropy ## pip dependency management doesn't work properly RUN pip install -r /python_packages ## Expose the ipython notebook port EXPOSE 8888 VOLUME /notebooks ## Run ipython notebook CMD /usr/bin/python /usr/local/bin/ipython notebook --ip=*--notebook-dir=/notebooks --pylab inline
21. Fig db: image: postgres:9.3 broker: image: dockerfile/rabbitmq worker: build: django_kat command: celery worker -A django_kat -l INFO links: - broker - db - mail environment: - DJANGO_SETTINGS_MODULE=django_kat.settings.container - C_FORCE_ROOT=true - SECRET_KEY="doesn't matter" django: build: django_kat command: uwsgi --socket django_kat.sock --module django_kat.wsgi links: - db - broker - mail volumes: - /code volumes_from: - worker viewer: image: gijzelaerr/cyberska_viewer volumes_from: - worker environment: - REDIRECT_URI ports: - 8081:80 - 8080:8080 https://github.com/ska-sa/ceiling-kat/blob/master/fig.yml
22. Docker hub https://registry.hub.docker.com/repos/radioastro/ upload docker image automated build from github repo 22
23. 2 use cases for docker software delivery build software solutions 23
24. Platforms 24
25. OSX
26. packaging https://launchpad.net/~radio-astro/+archive/u http://github.com/radio-astro/docker-images http://github.com/radio-astro/packaging
27. $ docker run -dP radioastro/notebook 27
28. RATT Online Deconvolved Radio Image Generation Using Eclectic / Exciting / Experimental / Exotic /Extraordinary / Existential / Ephemeral / Esoteric Software 28 R.O.D.R.I.G.U.E.S.
29. 29 Rodrigues
31. Problems security - trusted users only
32. Upcoming features https://github.com/docker/machine https://github.com/docker/swarm docker compose (CANCELLED - just fig)
33. Tips & tricks keep containers small Don’t build VM’s Make single purpose containers and link them together (fig) Use environment variables to configure containers Or mount config files in them docker exec is really useful 33
34. Thank You. 34 @gijzelaerr - http://pythonic.nl