Radio Astronomy Reduction Pipelines Using Python and Various Container Technologies by Athanaseus Ramaila

7b0645f018c0bddc8ce3900ccc3ba70c?s=47 Pycon ZA
October 11, 2019

Radio Astronomy Reduction Pipelines Using Python and Various Container Technologies by Athanaseus Ramaila

Modern-era radio interferometry software packages have improved enough to allow astronomers to start to address some of the calibration and imaging issues that are posed by the next generation radio interferometers, particularly the MeerKAT, LOFAR, ASKAP and Square Kilometre Array (SKA). I will present a framework that uses a platform-independent scripting tool called Stimela that allows the easy creation of astronomical data reduction pipelines using python and any of the supported (Docker, Podman, Singularity and uDocker) container technology of choice. In this framework, radio interferometry related tasks such as data synthesis, calibration and imaging are executed in containers. In fact, within this framework, the packages that perform these tasks are mostly Python modules. The primary aims are to provide the following services to the Radio Astronomy community: a) A user-friendly environment that gives general users easy access to novel radio interferometry packages. b) Simplified installation and production deployment. I will also discuss other great benefits to this deployment model such as providing a fixed production environment and reproducibility of scientific results.

7b0645f018c0bddc8ce3900ccc3ba70c?s=128

Pycon ZA

October 11, 2019
Tweet

Transcript

  1. Radio Astronomy Data Reduction Pipelines Using A Platform Independent Framework

    on behalf of the Radio Astronomy Research Group & MeerKATHI Group Athanaseus Ramaila
  2. None
  3. Electromagnetic wave

  4. None
  5. None
  6. None
  7. None
  8. Fact Sheet: • 64 Offset Gregorian steerable dishes 13.5 m

    in diameter • Baselines range between 7.7 km - 29.3 m • L-Band (856 - 1712 MHz) 208 kHz, 26 kHz, 3 kHz per chan correlator (Booth, R. S. et al. 2009)
  9. A Scientific Workflow Wishlist Level 1 Level 2 Level 3

    Easy to install Documented Scalable Easy to use Portable Smart with resources Reproducibility Configurable Quality Assessment Credit: S. Makhathini
  10. Stimela https://github.com/SpheMakh/Stimela

  11. None
  12. None
  13. None
  14. None
  15. FROM ubuntu:18.04 MAINTAINER gijsmolenaar@gmail.com ADD docker-apt-install /usr/local/bin ADD docker-apt-upgrade /usr/local/bin

    RUN docker-apt-upgrade RUN docker-apt-install \ software-properties-common RUN add-apt-repository -y \ ppa:kernsuite/kern-dev RUN add-apt-repository -y multiverse RUN add-apt-repository -y restricted Dockerfiles
  16. FROM ubuntu:18.04 MAINTAINER gijsmolenaar@gmail.com ADD docker-apt-install /usr/local/bin ADD docker-apt-upgrade /usr/local/bin

    RUN docker-apt-upgrade RUN docker-apt-install \ software-properties-common RUN add-apt-repository -y \ ppa:kernsuite/kern-dev RUN add-apt-repository -y multiverse RUN add-apt-repository -y restricted FROM kernsuite/base:5 RUN docker-apt-install \ python-pip git wget xvfb RUN pip install -U pip RUN pip install pyyaml COPY xvfb.init.d /etc/init.d/xvfb RUN chmod 755 /etc/init.d/xvfb RUN chmod 777 /var/run ENV DISPLAY :99 Dockerfiles
  17. FROM ubuntu:18.04 MAINTAINER gijsmolenaar@gmail.com ADD docker-apt-install /usr/local/bin ADD docker-apt-upgrade /usr/local/bin

    RUN docker-apt-upgrade RUN docker-apt-install \ software-properties-common RUN add-apt-repository -y \ ppa:kernsuite/kern-dev RUN add-apt-repository -y multiverse RUN add-apt-repository -y restricted FROM kernsuite/base:5 RUN docker-apt-install \ python-pip git wget xvfb RUN pip install -U pip RUN pip install pyyaml COPY xvfb.init.d /etc/init.d/xvfb RUN chmod 755 /etc/init.d/xvfb RUN chmod 777 /var/run ENV DISPLAY :99 FROM stimela/base:1.2.0 RUN docker-apt-install aoflagger Dockerfiles
  18. None
  19. AimFast RFInder RAGAVI RadioPadre

  20. SoFiA AimFast SHARPener SunBlocker RFInder AOFlagger RAGAVI RadioPadre

  21. SoFiA AimFast SHARPener SunBlocker RFInder AOFlagger RAGAVI RadioPadre TRIColour Meqtrees

    WSClean DDFacet PyBDSF +++ CubiCal CrystalBall
  22. None
  23. Stimela Structure Credit: S. Makhathini KERN Suite PPA PyPI

  24. Stimela Structure import stimela INPUT = "input" OUTPUT = "output"

    MSDIR = "msdir" recipe = stimela.Recipe("Plot table", ms_dir=MSDIR) table = "meerkathi-wlm-1gc1.B0" recipe.add("cab/ragavi", "gain_plotter", input=INPUT, output=OUTPUT,label="ragavi::ragavi", { "table" : "{:s}:output".format(table), "field" : 0, "gaintype": 'B', "htmlname": "meerkathi" }) recipe.run()
  25. Drasticali: A wrapper around Stimela https://github.com/Athanaseus/drasticali/wiki

  26. None
  27. None
  28. MeerKATHI - A pipeline to reduce radio interferometry data -

    Configured using a YML file - Easy to install and use - Documented https://meerkathi.readthedocs.io/en/latest/
  29. MeerKATHI Team Back: Daniel Molnar, Benjamin Hugo, Simon Perkins, Filippo

    Maccagni, Lexi Andati, Mpati Ramatsoku, Athanaseus Ramaila, Gyula I. G. Józsa, Kshitij Thorat Front: Paolo Serra, Sphesihle Makhathini, Dane Kleiner, Oleg Smirnov
  30. MeerKATHI Workflow Data Ingest Flagging Cross-Calibration Line Imaging Continuum Subtraction

    Self-Calibration Source Finding Source characterisation Reports Product Storage
  31. Credit: K. Thorat

  32. Achieved Scientific Workflow Wishlist Level 1 Level 2 Level 3

    Easy to install Documented Scalable Easy to use Portable Smart with resources Reproducibility Configurable Quality Assessment
  33. Circinus galaxy Thorat et al. 2019 Serra et al. 2019

    Fornax A 15′ Maccagni et al. 2019 Fornax A 15′ Ramatsoku et al. 2019 JO206 15′
  34. Summary MeerKAT is changing our view of the radio sky

    Automated data pipelines are the way Reproducible results
  35. Summary MeerKAT is changing our view of the radio sky

    Automated data pipelines are the way Reproducible results Stay tuned for the MeerKATHI public release!! meerkathi@googlegroups.com https://www.ska.ac.za
  36. Thank You! This project has received funding from the European

    Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 679627; project name FORNAX) and from a grant from the Italian Ministry of Foreign Affairs and International Cooperation (MAECI Grant Number ZA18GR02) and the South African Department of Science and Technology’s National Research Foundation (DST-NRF Grant Number 113121) as part of the ISARP Joint Research Scheme.