RaaS: Reproducibility as a Service by docker container on NIG supercomputer

Reproducible or Die

Transcript

1. Project RaaS: Reproducibility as a Service
2. Apache Mesos + Marathon + Chronos manager Node Node Node Node Node 1VCMJD1SJWBUF
   %PDLFS
   3FHJTUSZ %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage Basic Ideas and Design post post/get scp push pull run mount
3. Apache Mesos + Marathon + Chronos manager Node Node Node Node Node %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage %PDLFSGJMFT XPSLGMPXKTPO %BUB
4. workflow.json • JSON format configuration file • Describe a workflow contains multiple steps • 1 container for 1 app • Include directory to be mounted on containers
5. • JSON format configuration file • Describe a workflow contains multiple steps • 1 container for 1 app • Include directory to be mounted on containers { “user”: “inutano”, “project”: “exome”, “volume”: “/data”, “workflow”: [ {“image”:“awk”, “args”:[“1”,”10”], “cpu”:1, “memory”:”500M”}, {“image”:“bwa/bwa”, “args”:[“-v”]}, “cpu”:8, “memory”:”8G”, {“image”:”gatk/gatk”, “cpu”:”2”, “memory”: “4G”} ] }
6. workflow.json • Use REST API to put/get data to run workflow • $ curl -XPUT <uri> -d ‘<json>’ • create/read/update/delete workflow via API • URI should be unique for workflow/project/user • http://sc.ddbj.jp/user/project/workflow • Require Authentication manager
7. • Connect to Mesos, Marathon, Chronos to manage resources and running containers • Mesos mounts a directory to containers as described in workflow file • Kill containers depend on the running time, etc. manager Mesos
8. • Cost-Efficiency • ~5mins to prepare data analysis environments • Scalability • Parallel execution without env dependencies • Reproducibility • Manuscript = Raw data + workflow.json + Containers Benefit
9. Mesos + Marathon + Chronos manager Node %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node 1VCMJD
   4VQFSDPNQVUFS 1SJWBUF
   1$
   $MVTUFS $PNNFSDJBM
   $MPVE local manager SFRVJSF
   
   
   IJHI
   QFSGPSNBODF
   OFUXPSL
   
   
   PS
   NVMUJQMF
   DPQJFT
   UP
   SFEVDF
   
   
   EBUB
   USBOTGFS
   UJNF Future Plan
10. Project RaaS Tazro Inutano Ohta ! twitter.com/iNut github.com/inutano