RaaS: Reproducibility as a Service by docker container on NIG supercomputer

RaaS: Reproducibility as a Service by docker container on NIG supercomputer

Reproducible or Die

991f3366d9cc17386e6a66ef4abc6dbc?s=128

Tazro Inutano Ohta

January 22, 2015
Tweet

Transcript

  1. Project RaaS: Reproducibility as a Service

  2. Apache Mesos + Marathon + Chronos manager Node Node Node

    Node Node 1VCMJD1SJWBUF %PDLFS3FHJTUSZ %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage Basic Ideas and Design post post/get scp push pull run mount
  3. Apache Mesos + Marathon + Chronos manager Node Node Node

    Node Node %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage %PDLFSGJMFT XPSLGMPXKTPO %BUB
  4. workflow.json • JSON format configuration file • Describe a workflow

    contains multiple steps • 1 container for 1 app • Include directory to be mounted on containers
  5. • JSON format configuration file • Describe a workflow contains

    multiple steps • 1 container for 1 app • Include directory to be mounted on containers { “user”: “inutano”, “project”: “exome”, “volume”: “/data”, “workflow”: [ {“image”:“awk”, “args”:[“1”,”10”], “cpu”:1, “memory”:”500M”}, {“image”:“bwa/bwa”, “args”:[“-v”]}, “cpu”:8, “memory”:”8G”, {“image”:”gatk/gatk”, “cpu”:”2”, “memory”: “4G”} ] }
  6. workflow.json • Use REST API to put/get data to run

    workflow • $ curl -XPUT <uri> -d ‘<json>’ • create/read/update/delete workflow via API • URI should be unique for workflow/project/user • http://sc.ddbj.jp/user/project/workflow • Require Authentication manager
  7. • Connect to Mesos, Marathon, Chronos to manage resources and

    running containers • Mesos mounts a directory to containers as described in workflow file • Kill containers depend on the running time, etc. manager Mesos
  8. • Cost-Efficiency • ~5mins to prepare data analysis environments •

    Scalability • Parallel execution without env dependencies • Reproducibility • Manuscript = Raw data + workflow.json + Containers Benefit
  9. Mesos + Marathon + Chronos manager Node %PDLFSGJMFT XPSLGMPXKTPO %BUB

    Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node 1VCMJD4VQFSDPNQVUFS 1SJWBUF1$$MVTUFS $PNNFSDJBM$MPVE local manager SFRVJSF IJHIQFSGPSNBODFOFUXPSL PSNVMUJQMFDPQJFTUPSFEVDF EBUBUSBOTGFSUJNF Future Plan
  10. Project RaaS Tazro Inutano Ohta ! twitter.com/iNut github.com/inutano