Pro Yearly is on sale from $80 to $50! »

Data reduction pipelines using Docker for the MeerKAT telescope

Data reduction pipelines using Docker for the MeerKAT telescope

978e79ad01185b39efcfca1482f0f819?s=128

Gijs Molenaar

July 21, 2016
Tweet

Transcript

  1. THE MEERKAT TELESCOPE DATA REDUCTION USING DOCKER FOR Software Circus

    Amsterdam July 17 2016 Gijs Molenaar http://pythonic.nl
  2. WHO IS GIJS • SCIENTIFIC SOFTWARE ENGINEER • BACKGROUND IN

    AI • AMSTERDAM • CAPE TOWN • RADIO TELESCOPES
  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. MEERKAT TELESCOPE • 64 dishes • Raw data 4096 Gb/s

    • correlation • flagging • imaging • calibration
  10. None
  11. None
  12. None
  13. THE (IMAGING) SOFTWARE • Custom made software • developed by

    expert scientists • Cutting edge • Bleeding edge • Fragile • ‘Expensive’ to modify
  14. THE PROBLEM Running all this complex scientific software

  15. RADIO ASTRONOMY SOFTWARE SUITE KERN • Centralise the agony •

    Frustration for one person, the packager • A repository of commonly used radio astronomy software • Based on Ubuntu (16.04) • http://kernsuite.info
  16. None
  17. A B C C C fin Pipelines

  18. A B C C C fin

  19. KLIKO SCIENTIFIC COMPUTE CONTAINER FORMAT

  20. KLIKO • specification and library • based on docker •

    See a container as a compute unit • Consuming and producing file based data based on parameters • Abstract interface • easy to klikonize existing container • http://kliko.readthedocs.org
  21. HOW TO DO KLIKO • Read input from /input •

    Write output /output • /kliko.yml inside container defines parameters • /kliko script is entry point
  22. KLIKO RUNNER Actor that runs the container Read kliko.yml file

    Generate parameters (user input) Run container with parameters Connect correct input and output
  23. $ kliko-run radioastro/klikotest --help usage: kliko-run [-h] [--target_folder TARGET_FOLDER] --choice

    {second,first} --char CHAR [--float FLOAT] --file FILE --int INT image_name positional arguments: image_name optional arguments: -h, --help show this help message and exit --target_folder TARGET_FOLDER --choice {second,first} choice field (default: second) --char CHAR char field, maximum of 10 chars (default: empty) --float FLOAT float field (default: 0.0) --file FILE file field, this file will be put in /input in case of split io, /work in case of join io --int INT int field
  24. https://github.com/ska-sa/rodrigues

  25. FUNCTIONAL KLIKO CONTAINERS Chaining containers No side effects functional programming

    Caching results Implicit parallelisation airflow? luigi? something else? who?
  26. PROBLEMS WITH DOCKER Effectively giving root access GPU acceleration is

    crap Cached filesystem layers is just annoying Can’t combine containers A lot of new stuff I don’t care about (Windows)
  27. question time Gijs Molenaar @gijzelaerr http://pythonic.nl