Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RaaS: Reproducibility as a Service by docker container on NIG supercomputer

RaaS: Reproducibility as a Service by docker container on NIG supercomputer

Reproducible or Die

Tazro Inutano Ohta

January 22, 2015
Tweet

More Decks by Tazro Inutano Ohta

Other Decks in Science

Transcript

  1. Apache Mesos + Marathon + Chronos manager Node Node Node

    Node Node 1VCMJD1SJWBUF %PDLFS3FHJTUSZ %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage Basic Ideas and Design post post/get scp push pull run mount
  2. Apache Mesos + Marathon + Chronos manager Node Node Node

    Node Node %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage %PDLFSGJMFT XPSLGMPXKTPO %BUB
  3. workflow.json • JSON format configuration file • Describe a workflow

    contains multiple steps • 1 container for 1 app • Include directory to be mounted on containers
  4. • JSON format configuration file • Describe a workflow contains

    multiple steps • 1 container for 1 app • Include directory to be mounted on containers { “user”: “inutano”, “project”: “exome”, “volume”: “/data”, “workflow”: [ {“image”:“awk”, “args”:[“1”,”10”], “cpu”:1, “memory”:”500M”}, {“image”:“bwa/bwa”, “args”:[“-v”]}, “cpu”:8, “memory”:”8G”, {“image”:”gatk/gatk”, “cpu”:”2”, “memory”: “4G”} ] }
  5. workflow.json • Use REST API to put/get data to run

    workflow • $ curl -XPUT <uri> -d ‘<json>’ • create/read/update/delete workflow via API • URI should be unique for workflow/project/user • http://sc.ddbj.jp/user/project/workflow • Require Authentication manager
  6. • Connect to Mesos, Marathon, Chronos to manage resources and

    running containers • Mesos mounts a directory to containers as described in workflow file • Kill containers depend on the running time, etc. manager Mesos
  7. • Cost-Efficiency • ~5mins to prepare data analysis environments •

    Scalability • Parallel execution without env dependencies • Reproducibility • Manuscript = Raw data + workflow.json + Containers Benefit
  8. Mesos + Marathon + Chronos manager Node %PDLFSGJMFT XPSLGMPXKTPO %BUB

    Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node 1VCMJD4VQFSDPNQVUFS 1SJWBUF1$$MVTUFS $PNNFSDJBM$MPVE local manager SFRVJSF IJHIQFSGPSNBODFOFUXPSL PSNVMUJQMFDPQJFTUPSFEVDF EBUBUSBOTGFSUJNF Future Plan