Docker & radio astronomy

As a scientific software engineer in the field of radio astronomy, I'm involved in creating, improving and maintaining a broad range of tools used by scientists. These tools are used to process and analyse data coming from various radio telescopes, particularly LOFAR (LOw Frequency ARray) and SKA (Square Kilometre Array). Radio astronomy has a long and rich computing related history, and with that comes a big pile of poorly written, fragile, and badly maintained legacy code. This problem is not unique to radio astronomy but is a common in various scientific fields. Although no quick solution exists when it comes to re-writing lots of legacy code, Docker helps to contain this fragile software, simplifies the installation and ensures that the software works. In this talk I will discuss how Papino, our umbrella project for containing various radio astronomy libraries and RODRIGUES, an online radio telescope simulator leverage Docker.

Transcript

1. Docker & radio astronomy 1
2. Who Am I Gijs Molenaar Scientific Software Engineer in NL PhD student in SA MSc Artificial Intelligence 2 @gijzelaerr http://pythonic.nl
10. The problem 10
11. Scientist Q wants to crunch Data X with software Y on Platform Z 11
12. Scientist Q Smart person Self educated on the field of software engineer Gets the job done Good at duct tape programming 12
13. Data X Really big data Datasets of peta, exa bytes LOFAR - 50 GB/s SKA - 10 times global Internet traffic 13
14. Software Y 14 Complex math heavy software Written during PhD of scientist Q Often bad coding practices (Unit) tests? Version control? ‘Runs on my computer’
15. Platform Z 15
16. Solution until now 16
17. Solution until now 17 Compile everything by hand Still common practice (!)
20. Canonical launchpad 20
21. Papino 21 https://github.com/ska-sa/papino
22. $ docker run skasa/papino ipython notebook 22
23. Rodrigues - telescope simulator 23
29. science vs scalable reliable web apps Much less users Much more data High availability less important Security less important Performance very important Durability very important 29
30. Lessons learned Don’t build virtual machine images use small minimal containers use exec to examine problems inside container 30
31. Problems to solve / future work GPU acceleration Adaptation ‘Mixing’ containers (nixos?) Container size (South Africa) Container aware programming language / library? 31
32. Thank You. 32 @gijzelaerr - http://pythonic.nl