Bio In Docker Symposium, 9 November 2015, London.
http://core.brc.iop.kcl.ac.uk/wp-content/uploads/2015/02/BioInDockerAgenda.pdf
Paolo Di Tommaso - Notredame Lab Center for Genomic Regulation (CRG) Bio in Docker Symposium - 9 Nov 2015, LondonReproducible computationalpipelines with Docker andNextflow
View Slide
@PaoloDiTommaso Research software engineer Comparative Bioinformatics, Notredame Lab Center for Genomic Regulation (CRG)
WHAT THINGS MOST FRUSTRATE YOU ORLIMIT YOUR ABILITY TO CARRY OUTBIOINFORMATICS ANALYSIS?*• Ability to compile/run the average software suite. • Diversity and complexity of software deployments. • Poor and/or incomplete documentation. • Installation of different software, packages,... and getting them towork on different platforms. • Lack of truly standard file formats. • Time installing software. • Lack of computing resources (cpus, memory, storage, etc).* available at https://goo.gl/TF9TMj
To replicate the result of a typical computational biology paper requires 280 hours!
WHAT'S WRONG WITHCOMPUTATIONALWORKFLOWS?
COMPLEXITY• Dozens of dependencies (binary tools, compilers,libraries, system tools, etc) • Experimental nature of academic SW tends to bedifficult to install, configure and deploy • Heterogeneous executing platforms and systemarchitecture (laptop→supercomputer)
CONTAINERS ARE THE THIRD BIG WAVE IN VIRTUALISATION TECHNOLOGY
VM VSCONTAINER
BENEFITS• Smaller images (~100MB) • Fast instantiation time (~1sec) • Almost native performance • Easy to build, publish, share and deploy • Transparent build process
HostPACKAGING A WORKFLOWDocker imageBinary toolsWorkflow scriptsConfig fileCompilersLibrariesEnvironment
SCALING OUT. . . .
CONTAINERS ORCHESTRATION• Swarm • Fleet • Kubernetes • Mesos
NOT THE RIGHT ANSWERFOR COMPUTATIONALPIPELINES
SERVICES ORCHESTRATION ≠ TASKS SCHEDULING
OUR SOLUTIONNEXTFLOWHost file systemRegistry
DOCKER AT CRGNextflowConfig filePipeline scriptdocker registryhead nodeUniva grid engine
PROS• Dead easy deployment procedure • Self-contained and precise controlled runtime • Rapidly reproduce any former configuration • Consistent results over time and across differentplatforms
CONS• Requires a modern Linux kernel (≥3.10) • Security concerns • Containers/images cleanup
WHAT ABOUTPERFORMANCE?
BENCHMARK** Di Tommaso P, Palumbo E, Chatzou M, Prieto P, Heuer ML, Notredame C. (2015) The impact of Docker containers on the performance of genomic pipelines. PeerJ 3:e1273 https://dx.doi.org/10.7717/peerj.1273
• DSL (domain specify lang.) on top of JVM • High-level parallelisation model • Configurable executors target multiple platforms
RATIONALE• Make workflows portable across differentcomputational environments • Simplify deployment and enable reproducibility • Reuse any existing piece of SW (tools, scripts, etc)
DATAFLOW PROGRAMMINGABXYC D
process foo { ! input: val str from 'Hello' ! output: file 'my_file' into result ! script: """ echo $str world! > my_file """ } !PROCESS DEFINITION
WHAT A SCRIPT LOOKS LIKE!sequences = Channel.fromPath("/data/sample.fasta") process blast { input: file 'in.fasta' from sequences output: file 'out.txt' into blast_result !""" blastp -‐query in.fasta -‐outfmt 6 | cut -‐f 2 | \ blastdbcmd -‐entry_batch -‐ > out.txt """ } !process align { input: file all_seqs from blast_result output: file align_result !""" t_coffee $all_seqs 2>&-‐ | tee align_result """ } !blast_result.collectFile(name: 'final_alignment')
IMPLICIT PARALLELISM!sequences = Channel.fromPath("/data/*.fasta") process blast { input: file 'in.fasta' from sequences output: file 'out.txt' into blast_result !""" blastp -‐query in.fasta -‐outfmt 6 | cut -‐f 2 | \ blastdbcmd -‐entry_batch -‐ > out.txt """ } !process align { input: file all_seqs from blast_result output: file align_result !""" t_coffee $all_seqs 2>&-‐ | tee align_result """ } !blast_result.collectFile(name: 'final_alignment')
IMPLICIT PARALLELISMBLASTT-COFFEEBLASTT-COFFEET-COFFEEBLASTsample.fastasample.fastasample.fastaalignment
BENEFITS• High-level declarative parallelisation model • Portable across different platforms • Isolates task dependencies with Docker containers
CONFIGURATION FILE!process { container = 'your/image:latest' executor = 'sge' queue = 'cn-‐el6' memory = '10GB' cpus = 8 time = '2h' }
DEPLOYMENT MODES• Local execution • Grid engine / batch scheduler • Distributed execution (embedded cluster) • Managed cloud (ClusterK, DNAnexus) • AWS cloud platform
LOCAL EXECUTIONnextflow!procs!procsfile systemPOSIX procshost
GRID ENGINEnextflowlogin nodeNFScluster nodecluster nodecluster nodecluster nodebatch schedulersubmit taskscluster node
SUPPORTED PLATFORMS35
DISTRIBUTED MODELogin nodeNFS/LustreJob requestcluster nodecluster nodeJob wrapper! #!/bin/bash #$ -‐q #$ -‐pe ompi #$ -‐l virtual_free= mpirun nextflow run -‐with-‐mpi HPC clusternextflow clusternextflow drivernextflow workernextflow workernextflow workerApache Ignite
AWS CLOUDNextflow driverEC2 nodeS3 / EFSnextflow workersElastic Load balancersubmit tasksnextflow workersNextflow workersDocker registry AWS ECREC2 spot instancesApache IgniteApache Ignite
MANAGED CLOUDNextflowCirrus schedulerEC2 spotEC2 spotEC2 spotAmazon S3Submit tasksConfigure instances Execute tasksPipeline scriptCLUSTERK
DEMO
WHO IS USING NEXTFLOW?• Campagne Lab, Weill Medical College of Cornell University • Center for Biotechnology, Bielefeld University • Genetic Cancer group, International Agency Cancer Research • Guigo Lab, Center for Genomic Regulation • Medical genetics diagnostic, Oslo University Hospital • National Marrow Donor Program • Parasite Genomics, Sanger Institute • Sabeti Lab, Broad Institute • Veracyte Inc
FUTURE WORKShort term • Investigate support for Bioboxes • Support for Git LFS (large file storage) • Version 1.0 (first half 2016) Long term • Enhance processing capabilities with distributed caching anddata affinity. • Interoperability with YARN / Spark clusters / Common WL
CONCLUSION• Docker is a game-changer for workflowspackaging and deployment • Nextflow is a streaming oriented framework forcomputational workflows. • Docker + Nextflow = Reproducible self-contained pipelines.
THANKS
LINKSproject home http://nextflow.ioDocker benchmark https://peerj.com/articles/1273/Univa-CRG white paper http://goo.gl/lEPSe2this presentation https://speakerdeck.com/pditommaso
pditommaso
tenderlove
mongodb
lauravandoore
jakevdp
shpigford
mojombo
addyosmani
afnizarnur
jcasabona
aki_iinuma
revolveconf
tanoku