Share
Embed
Snakemake in the "Wild"
PyData Munich Meetup
Snakemake in the "Wild"
- Why should I use a workflow engine?
- What is Snakemake?
- Why should I use Snakemake?
- What can be done with Snakemake?
jfri3d
January 23, 2019
Transcript
56 slides
-
Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction
2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion -
Introduction Concept Snakemake Application Live Demo Conclusion Who am I...
• Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes -
Introduction Concept Snakemake Application Live Demo Conclusion Who am I...
• Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes -
Introduction Concept Snakemake Application Live Demo Conclusion Who am I...
• Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes -
Introduction Concept Snakemake Application Live Demo Conclusion Who am I...
• Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes -
Introduction Concept Snakemake Application Live Demo Conclusion Who am I...
• Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes -
Introduction Concept Snakemake Application Live Demo Conclusion Who am I...
• Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes -
Introduction Concept Snakemake Application Live Demo Conclusion Who am I...
• Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes -
Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010
2012 2014 2016 2018 2020 BSc Civil -
Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010
2012 2014 2016 2018 2020 BSc Civil Baird -
Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010
2012 2014 2016 2018 2020 BSc Civil Baird MSc Coastal -
Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010
2012 2014 2016 2018 2020 BSc Civil Baird MSc Coastal Deltares -
Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010
2012 2014 2016 2018 2020 BSc Civil Baird MSc Coastal Deltares TerraLoupe -
Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010
2012 2014 2016 2018 2020 BSc Civil Baird MSc Coastal Deltares TerraLoupe KI labs -
Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010
2012 2014 2016 2018 2020 TerraLoupe KI labs -
Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction
2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion -
Introduction Concept Snakemake Application Live Demo Conclusion Idea What about
doing this with QR codes? -
Introduction Concept Snakemake Application Live Demo Conclusion DAG • Directed
Acyclic Graph • Derived from Graph theory • Represents an execution plan • Nodes are jobs • Edges are dependencies DAG paths help differentiate between serial and parallel execution -
Introduction Concept Snakemake Application Live Demo Conclusion Execution How to
define, schedule and execute a DAG? • Workflow Engine • Workflow Manager • Workflow Management System • ... A workflow engine is a software application that manages processes. (Anonymous Wikipedia Contributor) -
Introduction Concept Snakemake Application Live Demo Conclusion Workflow Managers •
Snakemake • NextFlow • Ruffus • Toil • Airflow • Luigi • Pachyderm • Arvados • Taverna • Galaxy • Oozie • Cuneiform • Fireworks • Pypeline • Agave • BioDT • BPipe • Yabi • SciLuigi • Azkaban • Chronos -
Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction
2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion -
Introduction Concept Snakemake Application Live Demo Conclusion Overview The Snakemake
workflow management system is a tool to create reproducible and scalable data analyses. Workflows are described via a human readable, Python based language. They can be seamlessly scaled to server, cluster, grid and cloud environments, without the need to modify the workflow. -
Introduction Concept Snakemake Application Live Demo Conclusion Features • Reproducible
• Scalable • Implicit DAG Creation • Dynamic Rules • Wildcard Support • Human-readable Code • Remote Storage and Execution • Docker integration • Reporting • Logging • Temporary Files • Dry Run Execution • Parallel Execution • Detailed Run Statistics • Dynamic Resource Contraints • Rule Prioritization -
Introduction Concept Snakemake Application Live Demo Conclusion Workflow Definition Snakemake
workflows are essentially Python scripts extended by declarative code to define rules. Rules describe how to create output files from input files. -
Introduction Concept Snakemake Application Live Demo Conclusion Workflow Execution Snakemake
is primarily a command-line tool, so the snakemake executable is the primary way to execute, debug, and visualize workflows. Default execution with Snakefile in the current directory -
Introduction Concept Snakemake Application Live Demo Conclusion Workflow Execution Snakemake
is primarily a command-line tool, so the snakemake executable is the primary way to execute, debug, and visualize workflows. Dry run to test the workflow -
Introduction Concept Snakemake Application Live Demo Conclusion Workflow Execution Snakemake
is primarily a command-line tool, so the snakemake executable is the primary way to execute, debug, and visualize workflows. Parallel execution based on available cores -
Introduction Concept Snakemake Application Live Demo Conclusion Workflow Execution Snakemake
is primarily a command-line tool, so the snakemake executable is the primary way to execute, debug, and visualize workflows. Visualization of the entire DAG -
Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction
2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion -
Introduction Concept Snakemake Application Live Demo Conclusion Setup Python functions
with a clear dependency... -
Introduction Concept Snakemake Application Live Demo Conclusion Workflow - Input
String String input for searching GIPHY -
Introduction Concept Snakemake Application Live Demo Conclusion Workflow - GIPHY
JSON output from GIPHY -
Introduction Concept Snakemake Application Live Demo Conclusion Workflow - QR
Codes Embedded GIPHY links -
Introduction Concept Snakemake Application Live Demo Conclusion Execution DAG for
the previous workflow -
Introduction Concept Snakemake Application Live Demo Conclusion Execution DAG for
multiple GIPHY search terms -
Introduction Concept Snakemake Application Live Demo Conclusion Execution DAG for
multiple images and multiple GIPHY search terms -
Introduction Concept Snakemake Application Live Demo Conclusion Sample Output “Silly”
Nithish “Silly” Martin -
Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction
2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion -
Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction
2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion -
Introduction Concept Snakemake Application Live Demo Conclusion Key Takeaways •
Snakemake is an excellent workflow engine • It is flexible, scalable, reproducible, etc... • Snakemake is language and cloud agnostic • QR codes make great gifts and advent calendars -
Introduction Concept Snakemake Application Live Demo Conclusion Key Takeaways •
Snakemake is an excellent workflow engine • It is flexible, scalable, reproducible, etc... • Snakemake is language and cloud agnostic • QR codes make great gifts and advent calendars -
Introduction Concept Snakemake Application Live Demo Conclusion Key Takeaways •
Snakemake is an excellent workflow engine • It is flexible, scalable, reproducible, etc... • Snakemake is language and cloud agnostic • QR codes make great gifts and advent calendars -
Introduction Concept Snakemake Application Live Demo Conclusion Key Takeaways •
Snakemake is an excellent workflow engine • It is flexible, scalable, reproducible, etc... • Snakemake is language and cloud agnostic • QR codes make great gifts and advent calendars