PyGrunn 2016 - Kliko - scientific compute container

http://kliko.readthedocs.io

Transcript

1. KLIKO Scientific Compute Container Specification and Library Gijs Molenaar http://pythonic.nl
2. ME Scientific Software engineer Amsterdam & Cape Town Radio Telescopes
6. MeerKAT • Interferometry array • 64 dishes • 20 meter high
7. Cygnus A - O. Smirnov
10. IMAGING AND CALIBRATION Making the actual image Calibration - removing artefacts intrinsic to equipment Sort of similar to MRI imaging
11. SCIENTIFIC SOFTWARE Hard to install Fragile Package it up! Launchpad PPA
12. CONTAINERISE IT Self contained light weight ‘virtual machine’ Centralised agony Zero effort software install and distribution
13. WHAT’S NEXT? What else can we do using containerisation?
14. PIPELINES A B C C C fin
15. DOCKER DOCKER DOCKER A B C C C fin
16. BUT What does the output look like? What is in the input? How do I ‘configure’ the software
17. KLIKO - COMPUTE CONTAINER
19. Kliko container • Read input from /input • Write output /output • /kliko.yml inside container defines parameters • /kliko script is entry point
20. KLIKO RUNNER Actor that runs the container Read kliko.yml file Generate parameters (user input) Run container with parameters Connect correct input and output
21. $ kliko-run radioastro/klikotest --help usage: kliko-run [-h] [--target_folder TARGET_FOLDER] --choice {second,first} --char CHAR [--float FLOAT] --file FILE --int INT image_name positional arguments: image_name optional arguments: -h, --help show this help message and exit --target_folder TARGET_FOLDER --choice {second,first} choice field (default: second) --char CHAR char field, maximum of 10 chars (default: empty) --float FLOAT float field (default: 0.0) --file FILE file field, this file will be put in /input in case of split io, /work in case of join io --int INT int field
23. WHAT ELSE IS KILO Runner(s) validators Is kliko.yml valid? Are parameters matching kliko.yml file? Usable inside and outside container
24. FUNCTIONAL CONTAINERS functional programming Caching results Implicit parallelisation A B C C C fin
25. DOCKER VOLUME PLUGINS Amazon S3 OpenStack CEPH Google Compute
26. PROBLEMS WITH KILO No streaming (yet) Filesystem based, slow
27. PROBLEMS WITH DOCKER Effectively giving root access GPU acceleration is crap Cached filesystem layers is just annoying Container inheritance is a polytree
28. FUTURE WORK Streaming support kliko-compose?
29. KLIKO kliko.readthedocs.io https://github.com/gijzelaerr/kliko Python 2.7, 3.4+, PyPy 90% unittest coverage GPL2 Documented
30. QUESTIONS? http://pythonic.nl @gijzelaerr