Radio Astronomy Reduction Pipelines Using Python and Various Container Technologies by Athanaseus Ramaila

Modern-era radio interferometry software packages have improved enough to allow astronomers to start to address some of the calibration and imaging issues that are posed by the next generation radio interferometers, particularly the MeerKAT, LOFAR, ASKAP and Square Kilometre Array (SKA). I will present a framework that uses a platform-independent scripting tool called Stimela that allows the easy creation of astronomical data reduction pipelines using python and any of the supported (Docker, Podman, Singularity and uDocker) container technology of choice. In this framework, radio interferometry related tasks such as data synthesis, calibration and imaging are executed in containers. In fact, within this framework, the packages that perform these tasks are mostly Python modules. The primary aims are to provide the following services to the Radio Astronomy community: a) A user-friendly environment that gives general users easy access to novel radio interferometry packages. b) Simplified installation and production deployment. I will also discuss other great benefits to this deployment model such as providing a fixed production environment and reproducibility of scientific results.

Transcript

1. Radio Astronomy Data Reduction Pipelines Using A Platform Independent Framework on behalf of the Radio Astronomy Research Group & MeerKATHI Group Athanaseus Ramaila
3. Electromagnetic wave
8. Fact Sheet: • 64 Offset Gregorian steerable dishes 13.5 m in diameter • Baselines range between 7.7 km - 29.3 m • L-Band (856 - 1712 MHz) 208 kHz, 26 kHz, 3 kHz per chan correlator (Booth, R. S. et al. 2009)
9. A Scientific Workflow Wishlist Level 1 Level 2 Level 3 Easy to install Documented Scalable Easy to use Portable Smart with resources Reproducibility Configurable Quality Assessment Credit: S. Makhathini
10. Stimela https://github.com/SpheMakh/Stimela
15. FROM ubuntu:18.04 MAINTAINER gijsmolenaar@gmail.com ADD docker-apt-install /usr/local/bin ADD docker-apt-upgrade /usr/local/bin RUN docker-apt-upgrade RUN docker-apt-install \ software-properties-common RUN add-apt-repository -y \ ppa:kernsuite/kern-dev RUN add-apt-repository -y multiverse RUN add-apt-repository -y restricted Dockerfiles
16. FROM ubuntu:18.04 MAINTAINER gijsmolenaar@gmail.com ADD docker-apt-install /usr/local/bin ADD docker-apt-upgrade /usr/local/bin RUN docker-apt-upgrade RUN docker-apt-install \ software-properties-common RUN add-apt-repository -y \ ppa:kernsuite/kern-dev RUN add-apt-repository -y multiverse RUN add-apt-repository -y restricted FROM kernsuite/base:5 RUN docker-apt-install \ python-pip git wget xvfb RUN pip install -U pip RUN pip install pyyaml COPY xvfb.init.d /etc/init.d/xvfb RUN chmod 755 /etc/init.d/xvfb RUN chmod 777 /var/run ENV DISPLAY :99 Dockerfiles
17. FROM ubuntu:18.04 MAINTAINER gijsmolenaar@gmail.com ADD docker-apt-install /usr/local/bin ADD docker-apt-upgrade /usr/local/bin RUN docker-apt-upgrade RUN docker-apt-install \ software-properties-common RUN add-apt-repository -y \ ppa:kernsuite/kern-dev RUN add-apt-repository -y multiverse RUN add-apt-repository -y restricted FROM kernsuite/base:5 RUN docker-apt-install \ python-pip git wget xvfb RUN pip install -U pip RUN pip install pyyaml COPY xvfb.init.d /etc/init.d/xvfb RUN chmod 755 /etc/init.d/xvfb RUN chmod 777 /var/run ENV DISPLAY :99 FROM stimela/base:1.2.0 RUN docker-apt-install aoflagger Dockerfiles
19. AimFast RFInder RAGAVI RadioPadre
20. SoFiA AimFast SHARPener SunBlocker RFInder AOFlagger RAGAVI RadioPadre
21. SoFiA AimFast SHARPener SunBlocker RFInder AOFlagger RAGAVI RadioPadre TRIColour Meqtrees WSClean DDFacet PyBDSF +++ CubiCal CrystalBall
23. Stimela Structure Credit: S. Makhathini KERN Suite PPA PyPI
24. Stimela Structure import stimela INPUT = "input" OUTPUT = "output" MSDIR = "msdir" recipe = stimela.Recipe("Plot table", ms_dir=MSDIR) table = "meerkathi-wlm-1gc1.B0" recipe.add("cab/ragavi", "gain_plotter", input=INPUT, output=OUTPUT,label="ragavi::ragavi", { "table" : "{:s}:output".format(table), "field" : 0, "gaintype": 'B', "htmlname": "meerkathi" }) recipe.run()
25. Drasticali: A wrapper around Stimela https://github.com/Athanaseus/drasticali/wiki
28. MeerKATHI - A pipeline to reduce radio interferometry data - Configured using a YML file - Easy to install and use - Documented https://meerkathi.readthedocs.io/en/latest/
29. MeerKATHI Team Back: Daniel Molnar, Benjamin Hugo, Simon Perkins, Filippo Maccagni, Lexi Andati, Mpati Ramatsoku, Athanaseus Ramaila, Gyula I. G. Józsa, Kshitij Thorat Front: Paolo Serra, Sphesihle Makhathini, Dane Kleiner, Oleg Smirnov
30. MeerKATHI Workflow Data Ingest Flagging Cross-Calibration Line Imaging Continuum Subtraction Self-Calibration Source Finding Source characterisation Reports Product Storage
31. Credit: K. Thorat
32. Achieved Scientific Workflow Wishlist Level 1 Level 2 Level 3 Easy to install Documented Scalable Easy to use Portable Smart with resources Reproducibility Configurable Quality Assessment
33. Circinus galaxy Thorat et al. 2019 Serra et al. 2019 Fornax A 15′ Maccagni et al. 2019 Fornax A 15′ Ramatsoku et al. 2019 JO206 15′
34. Summary MeerKAT is changing our view of the radio sky Automated data pipelines are the way Reproducible results
35. Summary MeerKAT is changing our view of the radio sky Automated data pipelines are the way Reproducible results Stay tuned for the MeerKATHI public release!! meerkathi@googlegroups.com https://www.ska.ac.za
36. Thank You! This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 679627; project name FORNAX) and from a grant from the Italian Ministry of Foreign Affairs and International Cooperation (MAECI Grant Number ZA18GR02) and the South African Department of Science and Technology’s National Research Foundation (DST-NRF Grant Number 113121) as part of the ISARP Joint Research Scheme.