TRAnsient Pipeline TRAP - EuroSciPy 2014

My talk about the radio astronomical TRAnsient detection Pipeline at EuroSciPy 2014.

http://www.transientskp.org
http://docs.transientskp.org/tkp/master
https://github.com/transientskp/banana

Transcript

1. TRAnsient Pipeline Gijs Molenaar - @gijzelaerr - http://pythonic.nl Euroscipy 2014
2. Me • Scientific Software Engineer • PhD student (South Africa) ! • MSc Artificial Intelligence ! • Anton Pannekoek Instituut • SKA - Square Kilometre Array
3. What I do • Build software for processing telescope output used by scientists ! • ‘Working on design patterns and software techniques for open and reproducible data reduction’
4. Today • What is TRAP ! • How it’s made ! • Work in progress
9. Jess Broderick SS433 Famous microquasar
10. J. McKean and M. Wise, ASTRON
11. Lasting only for a short time; impermanent Transient
12. 50 Gb/s
13. 50 Gb/s
14. Virtual Observatory
17. Source Extraction • Position • Frequency • Time • Intensity (flux)
18. source association • Associate in time • Associate in frequency
19. Data rate • Data cubes • Multiple frequencies • 500 - 5000 sources / s • Will grow in future
20. A lot of data • So quite a lot of calculations • move calculations to data • Minimize communication
21. SQL • Storing, manipulating and retrieving data • De facto standard • Structured data • Relational data
22. Example
23. monolithic queries • hard to read • hard to reason about • Hard to atomise • hard to test • Code reuse difficult • hard to maintain
26. Database backends • PostgreSQL ! • MonetDB
27. MonetDB SQLAlchemy Djonet • pip install sqlalchemy-monetdb • http://github.com/gijzelaerr/sqlalchemy-monetdb ! • pip install djonet • http://github.com/gijzelaerr/djonet
28. Parallelisation • IPython • Home brew ssh based • Celery • Multiprocessing • IPython?
29. Visualisation • banana • Django • Highcharts • Bootstrap • Astropy & aplpy
32. Development process • Github • Pull Requests (4 eyes policy) • Documentation • Unit testing (77% of lines now) • Automated testing (jenkins, travis) • Performance tests (also jenkins) • Hipchat • weekly (skype) meeting, scientists, programmers
36. Deployment • When released ! • pip installable • Vagrant (virtual machine) • Docker
37. Status • 2.0 almost ready • Paper almost published • After everything BSD licensed ! • Banana already open source • https://github.com/transientskp/banana
38. Future plans • Ipython notebook based interface • (semi) public REST interface
39. Questions? http://pythonic.nl - @gijzelaerr