BH14.14 Team Docker Wrap-up

Introduction of docker in Life Science from wrap up session of BioHackathon satellite meeting BH14.14 at Sapporo

Transcript

1. BH1414 Team Docker http://wiki.lifesciencedb.jp/mw/BH14.14/Docker
2. Problem #EnvSettingSucks #SourcecodeBuildSucks #InstallingManualSucks #ReadingMaterialsandMethodsSucks #IwriteSomePerlScripts #Sucks
3. Docker!
5. the world without Docker
6. github repo? github.com/user/supercoolaligner thanks, later I’ll give it a try.. made a new tool for seq alignment
7. the world with Docker
8. made a new tool for seq alignment docker image? registry.hub.docker.com/u/user/supercoolaligner docker run -it user/supercoolaligner fantastic! but what does this value mean? Got it
9. Mac OS CoreOS Containers on CoreOS on MacOS
10. Mac OS CoreOS # require git, homebrew macos~$ brew tap phinze/homebrew-cask macos~$ brew install brew-cask macos~$ brew cask install virtualbox vagrant macos~$ git clone https://github.com/coreos/ coreos-vagrant && cd coreos-vagrant macos~$ vagrant up && vagrant ssh core@core01~$ docker run -it inutano/cmatrix
11. Mac OS CoreOS Port forward & Volume mount MacOS<-> CoreOS via Vagrantfile CoreOS<->Docker container via docker command See : http://bonohu.jp/blog/2015/02/02/docker-on-coreos-on-mac/
12. One of the Linux Distribution Minimize for container running etcd/fleet for cluster/container management See: http://deeeet.com/writing/2014/11/17/coreos/
13. Manage VirtualMachine via CLI Infrastructure as Code: Vagrantfile Distribute VM by Vagrantbox
14. Immutable Infrastructure and Disposable Components
16. $ vagrant up (messed up) $ vagrant destroy
18. Still Problem Data persistence Docker on Production Scale-out / Docker on large infrastructure Networking between containers Workflow with multiple containers
19. Project RaaS: Reproducibility as a Service
20. Apache Mesos + Marathon + Chronos manager Node Node Node Node Node 1VCMJD1SJWBUF
    %PDLFS
    3FHJTUSZ %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage Basic Ideas and Design post post/get scp push pull run mount
21. Apache Mesos + Marathon + Chronos manager Node Node Node Node Node %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage %PDLFSGJMFT XPSLGMPXKTPO %BUB
22. workflow.json • JSON format configuration file • Describe a workflow contains multiple steps • 1 container for 1 app • Include directory to be mounted on containers
23. • JSON format configuration file • Describe a workflow contains multiple steps • 1 container for 1 app • Include directory to be mounted on containers { “user”: “inutano”, “project”: “exome”, “volume”: “/data”, “workflow”: [ {“image”:“awk”, “args”:[“1”,”10”], “cpu”:1, “memory”:”500M”}, {“image”:“bwa/bwa”, “args”:[“-v”]}, “cpu”:8, “memory”:”8G”, {“image”:”gatk/gatk”, “cpu”:”2”, “memory”: “4G”} ] }
24. workflow.json • Use REST API to put/get data to run workflow • $ curl -XPUT <uri> -d ‘<json>’ • create/read/update/delete workflow via API • URI should be unique for workflow/project/user • http://sc.ddbj.jp/user/project/workflow • Require Authentication manager
25. • Connect to Mesos, Marathon, Chronos to manage resources and running containers • Mesos mounts a directory to containers as described in workflow file • Kill containers depend on the running time, etc. manager Mesos
26. • Cost-Efficiency • ~5mins to prepare data analysis environments • Scalability • Parallel execution without env dependencies • Reproducibility • Manuscript = Raw data + workflow.json + Containers Benefit
27. Mesos + Marathon + Chronos manager Node %PDLFSGJMFT XPSLGMPXKTPO %BUB Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node Mesos + Marathon + Chronos manager Node Storage Node Node Node 1VCMJD
    4VQFSDPNQVUFS 1SJWBUF
    1$
    $MVTUFS $PNNFSDJBM
    $MPVE local manager SFRVJSF
    
    
    IJHI
    QFSGPSNBODF
    OFUXPSL
    
    
    PS
    NVMUJQMF
    DPQJFT
    UP
    SFEVDF
    
    
    EBUB
    USBOTGFS
    UJNF Future Plan
28. Project RaaS Tazro Inutano Ohta ! twitter.com/iNut github.com/inutano