Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Nextflow: a tool for deploying reproducible computational pipelines

21c5e4164ca1573516b6a378fc279df2?s=47 Paolo Di Tommaso
July 11, 2015
260

Nextflow: a tool for deploying reproducible computational pipelines

Nextflow is a framework and a DSL modelled around the UNIX pipe concept, that simplifies writing parallel
pipelines in a portable manner

21c5e4164ca1573516b6a378fc279df2?s=128

Paolo Di Tommaso

July 11, 2015
Tweet

Transcript

  1. Bioinformatics Open Source Conference 11 july 2015 - Dublin, Ireland

    A tool for deploying reproducible computational pipelines
  2. Paolo Di Tommaso Research software engineer ~ 5 years at

    Notredame Lab Comparative Bioinformatics Center for Genomic Regulation (CRG)
  3. WHAT IS NEXTFLOW A framework and fluent DSL modelled around

    the UNIX pipe concept, that simplifies writing parallel pipelines in a portable manner
  4. MOTIVATION • Fast workflow prototyping • Reuse any existing scripts/tools

    • High-level parallelisation model • Portable across platforms • Enables reproducibility
  5. HOW IT WORKS • The pipeline flow is defined in

    a declarative manner • A script is composed by several processes • A process is defined by a set of inputs/outputs and a script snippet to be executed
  6. process  foo  {   !      input:    

       val  str  from  'Hello'   !      output:        file  'my_file'  into  result   !      script:        """        echo  $str  world!  >  my_file        """   }   ! PROCESS DEFINITION
  7. DATAFLOW • Declarative computational model for concurrent processes • Processes

    wait for data, when an input set is ready the process is executed • They communicate by using dataflow variables i.e. async stream of data called channels • Parallelisation and tasks dependencies are implicitly defined by process in/out declarations
  8. REACTIVE NETWORK

  9. WHAT A SCRIPT LOOKS LIKE ! params.query  =  "$baseDir/data/sample.fa"  

    ! seq  =  Channel.fromPath(params.query)                           ! process  blast  {          input:          file  'seq.fa'  from  seq   !        output:          file  'out.txt'  into  result   !        script:          """          blastp  -­‐db  NR  -­‐query  seq.fa  -­‐outfmt  6  >  out.txt          """   }   ! result.print  {  it.text  }    
  10. $ nextflow run blast-test.nf ! N E X T F

    L O W ~ version 0.12.0 [3d/ec5c2e] Submitted process > blast (1) ! 1ycsB 1YCS:B 100.00 60 0 0 1 60 170 229 3e-42 131 1ycsB 1ABO:B 24.07 54 39 1 3 56 6 57 4e-05 28.5 1ycsB 1ABO:A 24.07 54 39 1 3 56 6 57 4e-05 28.5 1ycsB 1PHT:A 30.43 23 16 0 6 28 10 32 0.013 22.3 1vie 1VIE:A 100.00 51 0 0 1 51 12 62 1e-35 108 1pht 1PHT:A 100.00 80 0 0 1 80 5 84 1e-56 164 1pht 1YCS:B 30.43 23 16 0 6 28 175 197 0.015 23.5 1pht 1IHF:B 33.33 21 14 0 53 73 60 80 0.75 18.1 1pht 1IHT:H 32.00 25 17 0 40 64 175 199 4.0 16.5 1pht 1IHS:H 32.00 25 17 0 40 64 175 199 4.0 16.5 10
  11. $ nextflow run blast-test.nf --query 'data/*.fa' ! N E X

    T F L O W ~ version 0.12.0 [3d/ec5c2e] Submitted process > blast (2) [1f/277042] Submitted process > blast (3) [9d/b49472] Submitted process > blast (4) [4a/3c2d5e] Submitted process > blast (1) [61/7dc8f0] Submitted process > blast (5) ! 1ycsB 1YCS:B 100.00 60 0 0 1 60 170 229 3e-42 131 1ycsB 1ABO:B 24.07 54 39 1 3 56 6 57 4e-05 28.5 1ycsB 1ABO:A 24.07 54 39 1 3 56 6 57 4e-05 28.5 1ycsB 1PHT:A 30.43 23 16 0 6 28 10 32 0.013 22.3 1vie 1VIE:A 100.00 51 0 0 1 51 12 62 1e-35 108 1pht 1PHT:A 100.00 80 0 0 1 80 5 84 1e-56 164 1pht 1YCS:B 30.43 23 16 0 6 28 175 197 0.015 23.5 1pht 1IHF:B 33.33 21 14 0 53 73 60 80 0.75 18.1 1pht 1IHT:H 32.00 25 17 0 40 64 175 199 4.0 16.5 1pht 1IHS:H 32.00 25 17 0 40 64 175 199 4.0 16.5 11
  12. NFS PLATFORM AGNOSTIC cluster engine local executor Nextflow grid executor

    Nextflow *nix OS
  13. SUPPORTED PLATFORMS 13

  14. HOW TO MAKE REPRODUCIBILITY EASIER?

  15. WHY IS 
 SO HARD TO DEPLOY A PIPELINE?

  16. COMMON PROBLEMS • Many dependencies (scripts, tools, DB, etc) •

    Environment configuration • Frequent updates • Track changes and versions • The tragedy of absolute paths ...
  17. OUR SOLUTION • Manage a pipeline as a *self-contained* Github

    repository • It includes external scripts and environment configuration • Binary dependencies can be deployed with Docker containers
  18. A GITHUB REPO CAN BE USED TO DEPLOY A SELF-CONTAINED

    PIPELINE PROJECT
  19. None
  20. $ nextflow run nextflow-io/rnatoy -with-docker ! N E X T

    F L O W ~ version 0.14.3 Pulling nextflow-io/rnatoy ... downloaded from https://github.com/nextflow-io/rnatoy.git ! Launching 'nextflow-io/rnatoy' - revision: 9c61bf5ac5 [master] R N A T O Y P I P E L I N E ================================= genome : /User/../data/ggal_1_4885000_49020000.Ggal71.500bp.fa annotat : /User/../data/ggal_1_4885000_49020000.bed.gff pair1 : /User/../data/*_1.fq pair2 : /User/../data/*_2.fq [warm up] executor > local [02/b08c28] Submitted process > buildIndex (ggal_1_4885000_49020000.Ggal71) [ea/97d004] Submitted process > mapping (ggal_gut) [98/16c9e5] Submitted process > mapping (ggal_liver) [b5/38a0c7] Submitted process > makeTranscript (ggal_gut) [00/e5efd6] Submitted process > makeTranscript (ggal_liver) Saving: transcript_ggal_gut.gtf Saving: transcript_ggal_liver.gtf 20
  21. DISTRIBUTED MODEL Nextflow Pipeline script Config file

  22. VERSIONING • Any Git tag, branch and commit ID can

    be used to a revision identifier • This allows to execute any previous version in a consistent manner
  23. $ nextflow run nextflow-io/rnatoy -revision v1.0 ! N E X

    T F L O W ~ version 0.14.3 Launching 'nextflow-io/rnatoy' - revision: 0d0443d8f7 [v1.0] R N A T O Y P I P E L I N E ================================= [35/cb611b] Submitted process > prepareTranscriptome (1) [cd/239926] Submitted process > buildIndex (1) [c6/f6488d] Submitted process > mapping (2) [bc/b3ea76] Submitted process > mapping (1) [f4/8d4628] Submitted process > makeTranscript (1) [eb/92db7f] Submitted process > makeTranscript (2) Saving: transcript_ggal_alpha.gtf Saving: transcript_ggal_beta.gtf 23
  24. WHO IS USING NEXTFLOW? • Andrew Stewart, Veracyte Inc •

    Emilio Palumbo, Center for Genomic Regulation • Georgios Pappas, University of Brasilia • Lukas Jelonek, Justus-Liebig-Universität Gießen • Matthieu Foll, International Agency for Research on Cancer • Michael L Heuer, National Marrow Donor Program • Rémi Planel, University Claude Bernard Lyon 1 • Rob Syme, CCDM, Curtin University • Sascha Steinbiss, Sanger Institute • Simon Ye, Broad Institute • Tobias Sargeant, The Walter and Eliza Hall Institute
  25. FUTURE WORK Short term • Built-in low-latency cluster Medium term

    • Support for Bioboxes project • Support for Git Large File Storage • Support for Rkt containers Long term • Graphical editor • Support for CWL / WDL (?) • Support for YARN / Spark clusters
  26. KEEP IN TOCH Home page
 http://nextflow.io Github
 https://github.com/nextflow-io/nextflow Chat
 https://gitter.im/nextflow-io/nextflow