Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Docker & radio astronomy - containing fragile scientific software

978e79ad01185b39efcfca1482f0f819?s=47 Gijs Molenaar
January 24, 2015

Docker & radio astronomy - containing fragile scientific software

As a scientific software engineer in the field of radio astronomy, I'm involved in creating, improving and maintaining a broad range of tools used by scientists. These tools are used to process and analyse data coming from various radio telescopes, particularly LOFAR (LOw Frequency ARray) and SKA (Square Kilometre Array). Radio astronomy has a long and rich computing related history, and with that comes a big pile of poorly written, fragile, and badly maintained legacy code. This problem is not unique to radio astronomy but is a common in various scientific fields.

Although no quick solution exists when it comes to re-writing lots of legacy code, Docker helps to contain this fragile software, simplifies the installation and ensures that the software works. In this talk I will discuss how Papino, our umbrella project for containing various radio astronomy libraries and RODRIGUES, an online radio telescope simulator leverage Docker .

978e79ad01185b39efcfca1482f0f819?s=128

Gijs Molenaar

January 24, 2015
Tweet

Transcript

  1. Docker & radio astronomy 1

  2. Who Am I Gijs Molenaar Scientific Software Engineer in NL

    PhD student in SA MSc Artificial Intelligence 2 @gijzelaerr http://pythonic.nl
  3. Agenda Radio astronomy Problems in science What is Docker How

    we use it 3
  4. None
  5. 5

  6. 6

  7. 7

  8. 8

  9. 9

  10. Scientist Q wants to crunch Data X with software Y

    on Platform Z 10
  11. Scientist Q Smart person Self educated on the field of

    software engineer Gets the job done Good at duct tape programming 11
  12. Data X Really big data Datasets of peta, exa bytes

    LOFAR - 50 GB/s SKA - 10 times global Internet traffic 12
  13. Software Y 13 Complex math heavy software Written during PhD

    of scientist Q Often bad coding practices (Unit) tests? Version control? ‘Runs on my computer’
  14. What is docker? ‘Containing’ software Managing containers Connecting containers Deploying

    containers 14
  15. Like virtual machines But not no emulation Process isolation &

    virtual networking More like chroot/jail 15
  16. None
  17. Image vs container Image set of ‘layers’ read-only stateless build

    with Dockerfile or commit container container instantiation of image stateful
  18. Layered filesytem (unionFS) 18

  19. Dockerfile FROM ubuntu:14.04 MAINTAINER gijs@pythonic.nl ENV DEBIAN_FRONTEND noninteractive # enable

    universe, multiverse, restricted with world wide mirrors ADD apt.sources.list /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y RUN apt-get install -y software-properties-common ## add radio astro PPA (with source packages) RUN add-apt-repository -ys ppa:radio-astro/main RUN apt-get update https://github.com/radio-astro/docker-images 19
  20. Dockerfile 2 FROM radioastro/base ADD debian_packages / ADD ska_packages /

    ADD python_packages / # update deb cache RUN apt-get update # install standard debian packages RUN cat /debian_packages | xargs apt-get install -y # installed the packaged software from the PPA RUN cat /ska_packages | xargs apt-get install -y # install all python modules RUN pip install astropy ## pip dependency management doesn't work properly RUN pip install -r /python_packages ## Expose the ipython notebook port EXPOSE 8888 VOLUME /notebooks ## Run ipython notebook CMD /usr/bin/python /usr/local/bin/ipython notebook --ip=*--notebook-dir=/notebooks --pylab inline
  21. Fig db: image: postgres:9.3 broker: image: dockerfile/rabbitmq worker: build: django_kat

    command: celery worker -A django_kat -l INFO links: - broker - db - mail environment: - DJANGO_SETTINGS_MODULE=django_kat.settings.container - C_FORCE_ROOT=true - SECRET_KEY="doesn't matter" django: build: django_kat command: uwsgi --socket django_kat.sock --module django_kat.wsgi links: - db - broker - mail volumes: - /code volumes_from: - worker viewer: image: gijzelaerr/cyberska_viewer volumes_from: - worker environment: - REDIRECT_URI ports: - 8081:80 - 8080:8080 https://github.com/ska-sa/ceiling-kat/blob/master/fig.yml
  22. Docker hub https://registry.hub.docker.com/repos/radioastro/ upload docker image automated build from github

    repo 22
  23. 2 use cases for docker software delivery build software solutions

    23
  24. Platforms 24

  25. OSX

  26. packaging https://launchpad.net/~radio-astro/+archive/u http://github.com/radio-astro/docker-images http://github.com/radio-astro/packaging

  27. $ docker run -dP radioastro/notebook 27

  28. RATT Online Deconvolved Radio Image Generation Using Eclectic / Exciting

    / Experimental / Exotic /Extraordinary / Existential / Ephemeral / Esoteric Software 28 R.O.D.R.I.G.U.E.S.
  29. 29 Rodrigues

  30. 30

  31. Problems security - trusted users only

  32. Upcoming features https://github.com/docker/machine https://github.com/docker/swarm docker compose (CANCELLED - just fig)

  33. Tips & tricks keep containers small Don’t build VM’s Make

    single purpose containers and link them together (fig) Use environment variables to configure containers Or mount config files in them docker exec is really useful 33
  34. Thank You. 34 @gijzelaerr - http://pythonic.nl