Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyGrunn 2016 - Kliko - scientific compute container

PyGrunn 2016 - Kliko - scientific compute container

978e79ad01185b39efcfca1482f0f819?s=128

Gijs Molenaar

May 12, 2016
Tweet

Transcript

  1. KLIKO Scientific Compute Container Specification and Library Gijs Molenaar http://pythonic.nl

  2. ME Scientific Software engineer Amsterdam & Cape Town Radio Telescopes

  3. None
  4. None
  5. None
  6. MeerKAT • Interferometry array • 64 dishes • 20 meter

    high
  7. Cygnus A - O. Smirnov

  8. None
  9. None
  10. IMAGING AND CALIBRATION Making the actual image Calibration - removing

    artefacts intrinsic to equipment Sort of similar to MRI imaging
  11. SCIENTIFIC SOFTWARE Hard to install Fragile Package it up! Launchpad

    PPA
  12. CONTAINERISE IT Self contained light weight ‘virtual machine’ Centralised agony

    Zero effort software install and distribution
  13. WHAT’S NEXT? What else can we do using containerisation?

  14. PIPELINES A B C C C fin

  15. DOCKER DOCKER DOCKER A B C C C fin

  16. BUT What does the output look like? What is in

    the input? How do I ‘configure’ the software
  17. KLIKO - COMPUTE CONTAINER

  18. None
  19. Kliko container • Read input from /input • Write output

    /output • /kliko.yml inside container defines parameters • /kliko script is entry point
  20. KLIKO RUNNER Actor that runs the container Read kliko.yml file

    Generate parameters (user input) Run container with parameters Connect correct input and output
  21. $ kliko-run radioastro/klikotest --help usage: kliko-run [-h] [--target_folder TARGET_FOLDER] --choice

    {second,first} --char CHAR [--float FLOAT] --file FILE --int INT image_name positional arguments: image_name optional arguments: -h, --help show this help message and exit --target_folder TARGET_FOLDER --choice {second,first} choice field (default: second) --char CHAR char field, maximum of 10 chars (default: empty) --float FLOAT float field (default: 0.0) --file FILE file field, this file will be put in /input in case of split io, /work in case of join io --int INT int field
  22. None
  23. WHAT ELSE IS KILO Runner(s) validators Is kliko.yml valid? Are

    parameters matching kliko.yml file? Usable inside and outside container
  24. FUNCTIONAL CONTAINERS functional programming Caching results Implicit parallelisation A B

    C C C fin
  25. DOCKER VOLUME PLUGINS Amazon S3 OpenStack CEPH Google Compute

  26. PROBLEMS WITH KILO No streaming (yet) Filesystem based, slow

  27. PROBLEMS WITH DOCKER Effectively giving root access GPU acceleration is

    crap Cached filesystem layers is just annoying Container inheritance is a polytree
  28. FUTURE WORK Streaming support kliko-compose?

  29. KLIKO kliko.readthedocs.io https://github.com/gijzelaerr/kliko Python 2.7, 3.4+, PyPy 90% unittest coverage

    GPL2 Documented
  30. QUESTIONS? http://pythonic.nl @gijzelaerr