Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Enabling reproducible in-silico data analises with Nextflow

Enabling reproducible in-silico data analises with Nextflow

Paolo Di Tommaso

May 01, 2018
Tweet

More Decks by Paolo Di Tommaso

Other Decks in Research

Transcript

  1. ENABLING REPRODUCIBLE 
 IN-SILICO DATA ANALISES WITH NEXTFLOW Paolo Di

    Tommaso, CRG Wellcome Trust Sanger Institute, 1 May 2018, Cambridge
  2. WHO IS THIS CHAP? @PaoloDiTommaso Research software engineer Comparative Bioinformatics,

    Notredame Lab Center for Genomic Regulation (CRG) Author of Nextflow project
  3. AGENDA • The challenges with computational workflows • Nextflow main

    principles • Handling parallelisation and portability • Deployments scenarios • Comparison with other tools • Future plans
  4. GENOMIC WORKFLOWS • Data analysis applications to extract information from

    (large) genomic datasets • Embarrassingly parallelisation, can spawn 100s-100k jobs over distributed cluster • Mash-up of many different tools and scripts • Complex dependency trees and configuration → very fragile ecosystem
  5. Steinbiss et al., Companion parassite genome annotation pipeline, DOI: 10.1093/nar/gkw292

  6. To reproduce the result of a typical 
 computational biology

    paper
 requires 280 hours. ≈1.7 months!
  7. None
  8. THE SAME APPLICATION DEPLOYED IN DIFFERENT ENVIRONMENTS PRODUCES DIFFERENT RESULTS

    (!)
  9. Platform Amazon Linux Debian Linux Mac OSX Number of chromosomes

    36 36 36 Overall length (bp) 32,032,223 32,032,223 32,032,223 Number of genes 7,781 7,783 7,771 Gene density 236.64 236.64 236.32 Number of coding genes 7,580 7,580 7570 Average coding length (bp) 1,764 1,764 1,762 Number of genes with multiple CDS 113 113 111 Number of genes with known function 4,147 4,147 4,142 Number of t-RNAs 88 90 88 Comparison of the Companion pipeline annotation of Leishmania infantum genome executed across different platforms * * Di Tommaso P, et al., Nextflow enables computational reproducibility, Nature Biotech, 2017
  10. CHALLENGES • Reproducibility, replicate results over time • Portability, run

    across different platforms • Scalability ie. deploy big distributed workloads • Usability, streamline execution and deployment of complex workloads ie. remove complexity instead of adding new one • Consistency ie. track changes and revisions consistently for code, config files and binary dependencies
  11. PUSH-THE-BUTTON PIPELINES

  12. HOW? • Fast prototyping 㱺 custom DSL that enables tasks

    composition, simplifies most use cases + general purpose programming lang. for corner cases • Easy parallelisation 㱺 declarative reactive programming model based on dataflow paradigm, implicit portable parallelism • Self-contained 㱺 functional approach, a task execution is idempotent ie. cannot modify the state of other tasks + isolate dependencies with containers • Portable deployments 㱺 executor abstraction layer + deployment configuration from implementation logic
  13. Orchestration & Parallelisation Scalability & Portability Deployment & Reproducibility containers

    Git GitHub
  14. TASK EXAMPLE bwa mem reference.fa sample.fq \ | samtools sort

    -o sample.bam
  15. process align_sample { input: file 'reference.fa' from genome_ch file 'sample.fq'

    from reads_ch output: file 'sample.bam' into bam_ch script: """ bwa mem reference.fa sample.fq \ | samtools sort -o sample.bam """ } TASK EXAMPLE bwa mem reference.fa sample.fq \ | samtools sort -o sample.bam
  16. TASKS COMPOSITION process index_sample { input: file 'sample.bam' from bam_ch

    output: file 'sample.bai' into bai_ch script: """ samtools index sample.bam """ } process align_sample { input: file 'reference.fa' from genome_ch file 'sample.fq' from reads_ch output: file 'sample.bam' into bam_ch script: """ bwa mem reference.fa sample.fq \ | samtools sort -o sample.bam """ }
  17. DATAFLOW • Declarative computational model for parallel process executions •

    Processes wait for data, when an input set is ready the process is executed • They communicate by using dataflow variables i.e. async FIFO queues called channels • Parallelisation and tasks dependencies are implicitly defined by process in/out declarations
  18. HOW PARALLELISATION WORKS samples_ch = Channel.fromPath('data/sample.fastq') process FASTQC { input:

    file reads from samples_ch output: file 'fastqc_logs' into fastqc_ch """ mkdir fastqc_logs fastqc -o fastqc_logs -f fastq -q ${reads} """ }
  19. samples_ch = Channel.fromPath('data/*.fastq') process FASTQC { input: file reads from

    samples_ch output: file 'fastqc_logs' into fastqc_ch """ mkdir fastqc_logs fastqc -o fastqc_logs -f fastq -q ${reads} """ } HOW PARALLELISATION WORKS
  20. IMPLICIT PARALLELISM clustalo Channel.fromPath("data/*.fastq") clustalo FASTQC

  21. SUPPORTED PLATFORMS

  22. DEPLOYMENT SCENARIOS

  23. LOCAL EXECUTION • Common development scenario • Dependencies can be

    managed using a container runtime • Parallelisations is managed spawning posix processes • Can scale vertically using fat server / shared mem. machine nextflow OS local storage docker/singularity laptop / workstation
  24. CENTRALISED ORCHESTRATION computer cluster • Nextflow orchestrates workflow execution submitting

    jobs to a compute cluster eg. SLURM • It can run in the head node or a compute node • Requires a shared storage to exchange data between tasks • Ideal for corse-grained parallelisms NFS/Lustre cluster node cluster node cluster node cluster node submit jobs cluster node nextflow
  25. DISTRIBUTED ORCHESTRATION login node NFS/Lustre job request cluster node cluster

    node launcher wrapper nextflow cluster nextflow driver nextflow worker nextflow worker nextflow worker HPC cluster • A single job request allocates the desired computes nodes • Nextflow deploys its own embedded compute cluster • The main instance orchestrate the workflow execution • The worker instances execute workflow jobs (work stealing approach)
  26. KUBERNETES • Next generation native cloud clustering for containerised workloads

    • There's the need of workflow orchestration • Latest NF version includes a new command that streamline the workflow deployment to K8s
  27. K8S DEPLOYMENT

  28. PORTABILITY

  29. PORTABILITY process { executor = 'slurm' queue = 'my-queue' memory

    = '8 GB' cpus = 4 container = 'user/image' }
  30. PORTABILITY process { executor = 'awsbatch' queue = 'my-queue' memory

    = '8 GB' cpus = 4 container = 'user/image' }
  31. CONFIGURATION DECOUPLING 
 IS THE KEY TO PORTABLE DEPLOYMENTS

  32. DEMO!

  33. A QUICK COMPARISON

  34. GALAXY vs. NEXTFLOW • Command line oriented tool • Can

    incorporate any tool w/o any extra adapter • Fine control over tasks parallelisation • Scalability 100㱺1M jobs • One liner installer • Suited for production workflows + experienced bioinformaticians • Web based platform • Built-in integration with many tools and dataset • Little control over tasks parallelisation • Scalability 10㱺1K jobs • Complex installation and maintenance • Suited for training + not experienced bioinformaticians
  35. SNAKEMAKE vs. NEXTFLOW • Command line oriented tool • Push

    model • Can manage any data structure • Compute DAG at runtime • All major container runtimes • Built-in support for clusters and cloud • No (yet) support for sub-workflows • Built-in support for Git/GitHub, etc., manage pipeline revisions • Groovy/JVM based • Command line oriented tool • Pull model • Rules defined using file name patterns • Compute DAG ahead • Built-in support for Singularity • Custom scripts for cluster deployments • Support for sub-workflows • No support for source code management system • Python based
  36. CWL vs. NEXTFLOW • Language + app. runtime • DSL

    on top of a general purpose programming lang. • Concise, fluent (at least try to be!) • Community driven • Single implementation, quick iterations • Language specification • Declarative meta-language (YAML/JSON) • Verbose • Committee driven • Many vendors/implementations (and specification version)
  37. CONTAINERISATION

  38. CONTAINERISATION • Nextflow envisioned the use of software containers to

    fix computational reproducibility • Mar 2014 (ver 0.7), support for Docker • Dec 2016 (ver 0.23), support for Singularity Nextflow job job job
  39. SINGULARITY FEATURES Kurtzer et al. Singularity: Scientific containers for mobility

    of compute. PLoS ONE 12(5): e0177459
  40. BENCHMARK* * Di Tommaso P, Palumbo E, Chatzou M, Prieto

    P, Heuer ML, Notredame C. (2015) 
 The impact of Docker containers on the performance of genomic pipelines. PeerJ 3:e1273 
 https://dx.doi.org/10.7717/peerj.1273 container execution can have an impact on short running tasks ie. < 1min
  41. SINGULARITY BENCHMARK https://github.com/wresch/python_import_problem Singularity image format speeds up Python execution

    having many imports 
 from a shared file system !
  42. WHEN USE CONTAINERS? ALWAYS!

  43. BEST PRACTICES • Helps to isolate dependencies from dev or

    local deployment environment • Provides a reproducibles sandbox for third party users • Binary images preserve against software decay • Make it transparent ie. always include the Dockefile • Docker image format is de-facto standard, it can be executed by different runtime eg. Singularity, Shifter, uDocker, etc.
  44. ERROR RECOVERY • Each task outputs are saved in a

    separate directory • This allows to safely record interrupted executions discarding • Dramatically simplify debugging ! • Computing resources can be defined in a *dynamic* manner, so that a failing task can be automatically re- execute with more memory, longer timeout, etc.
  45. EXECUTION REPORT

  46. EXECUTION REPORT

  47. EXECUTION TIMELINE

  48. DAG VISUALISATION

  49. EDITORS !

  50. WHAT'S NEXT

  51. IMPROVEMENTS • Built-in support for Bioconda recipies • Better meta-data

    and provenance handling • Workflow composition aka sub-workflows • More clouds support ie. Azure and GCP
  52. APACHE SPARK • Native support for Apache Spark clusters and

    execution model • Allow hybrid Nextflow and Spark applications • Mix the best of the two worlds, Nextflow for legacy tools/corse grain parallelisation and Spark for fine grain/distributed execution eg. GATK4
  53. • Partecipate in Cloud Work Stream working group • TES:

    Task Execution API (working prototype) • WES: Workflow Execution API • Enable interoperability with GA4GH complaint platforms eg. Cancer Genomics Cloud and Broad FireCloud
  54. WHO IS USING NEXTFLOW?

  55. • Community effort to collect production ready analysis pipelines built

    with Nextflow • Initially supported by SciLifeLab, QBiC and A*Star Genome Institute Singapore • https://nf-core.github.io Alexander 
 Peltzer Phil Ewels Andreas Wilm
  56. CONCLUSION • Data analysis reproducibility is hard and it's often

    underestimated. • Nextflow does not provide a magic solution but enables best-practices and provide support for community and industry standards. • It strictly separates the application logic from the configuration and deployment logic, enabling self-contained workflows. • Applications can be easily deployed across different environment in a reproducible manner with a single command. • The functional/reactive model allows applications to scale to millions of jobs with ease.
  57. ACKNOWLEDGMENT Evan Floden Emilio Palumbo Cedric Notredame Notredame Lab, CRG

    http://nextflow.io