BH14.14 Team Docker Wrap-up

BH14.14 Team Docker Wrap-up

Introduction of docker in Life Science from wrap up session of BioHackathon satellite meeting BH14.14 at Sapporo

991f3366d9cc17386e6a66ef4abc6dbc?s=128

Tazro Inutano Ohta

February 06, 2015
Tweet

Transcript

  1. BH1414 Team Docker http://wiki.lifesciencedb.jp/mw/BH14.14/Docker

  2. Problem #EnvSettingSucks #SourcecodeBuildSucks #InstallingManualSucks #ReadingMaterialsandMethodsSucks #IwriteSomePerlScripts #Sucks

  3. Docker!

  4. None
  5. the world without Docker

  6. github repo? github.com/user/supercoolaligner thanks, later I’ll give it a try..

    made a new tool for seq alignment
  7. the world with Docker

  8. made a new tool for seq alignment docker image? registry.hub.docker.com/u/user/supercoolaligner

    docker run -it user/supercoolaligner fantastic! but what does this value mean? Got it
  9. Mac OS CoreOS Containers on CoreOS on MacOS

  10. Mac OS CoreOS # require git, homebrew macos~$ brew tap

    phinze/homebrew-cask macos~$ brew install brew-cask macos~$ brew cask install virtualbox vagrant macos~$ git clone https://github.com/coreos/ coreos-vagrant && cd coreos-vagrant macos~$ vagrant up && vagrant ssh core@core01~$ docker run -it inutano/cmatrix
  11. Mac OS CoreOS Port forward & Volume mount MacOS<-> CoreOS

    via Vagrantfile CoreOS<->Docker container via docker command See : http://bonohu.jp/blog/2015/02/02/docker-on-coreos-on-mac/
  12. One of the Linux Distribution Minimize for container running etcd/fleet

    for cluster/container management See: http://deeeet.com/writing/2014/11/17/coreos/
  13. Manage VirtualMachine via CLI Infrastructure as Code: Vagrantfile Distribute VM

    by Vagrantbox
  14. Immutable Infrastructure and Disposable Components

  15. None
  16. $ vagrant up (messed up) $ vagrant destroy

  17. None
  18. Still Problem Data persistence Docker on Production Scale-out / Docker

    on large infrastructure Networking between containers Workflow with multiple containers
  19. Project RaaS: Reproducibility as a Service

  20. Apache Mesos + Marathon + Chronos manager Node Node Node

    Node Node 1VCMJD1SJWBUF %PDLFS3FHJTUSZ %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage Basic Ideas and Design post post/get scp push pull run mount
  21. Apache Mesos + Marathon + Chronos manager Node Node Node

    Node Node %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage %PDLFSGJMFT XPSLGMPXKTPO %BUB
  22. workflow.json • JSON format configuration file • Describe a workflow

    contains multiple steps • 1 container for 1 app • Include directory to be mounted on containers
  23. • JSON format configuration file • Describe a workflow contains

    multiple steps • 1 container for 1 app • Include directory to be mounted on containers { “user”: “inutano”, “project”: “exome”, “volume”: “/data”, “workflow”: [ {“image”:“awk”, “args”:[“1”,”10”], “cpu”:1, “memory”:”500M”}, {“image”:“bwa/bwa”, “args”:[“-v”]}, “cpu”:8, “memory”:”8G”, {“image”:”gatk/gatk”, “cpu”:”2”, “memory”: “4G”} ] }
  24. workflow.json • Use REST API to put/get data to run

    workflow • $ curl -XPUT <uri> -d ‘<json>’ • create/read/update/delete workflow via API • URI should be unique for workflow/project/user • http://sc.ddbj.jp/user/project/workflow • Require Authentication manager
  25. • Connect to Mesos, Marathon, Chronos to manage resources and

    running containers • Mesos mounts a directory to containers as described in workflow file • Kill containers depend on the running time, etc. manager Mesos
  26. • Cost-Efficiency • ~5mins to prepare data analysis environments •

    Scalability • Parallel execution without env dependencies • Reproducibility • Manuscript = Raw data + workflow.json + Containers Benefit
  27. Mesos + Marathon + Chronos manager Node %PDLFSGJMFT XPSLGMPXKTPO %BUB

    Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node 1VCMJD4VQFSDPNQVUFS 1SJWBUF1$$MVTUFS $PNNFSDJBM$MPVE local manager SFRVJSF IJHIQFSGPSNBODFOFUXPSL PSNVMUJQMFDPQJFTUPSFEVDF EBUBUSBOTGFSUJNF Future Plan
  28. Project RaaS Tazro Inutano Ohta ! twitter.com/iNut github.com/inutano