Snakemake in the "Wild"

PyData Munich Meetup

Snakemake in the "Wild"
- Why should I use a workflow engine?
- What is Snakemake?
- Why should I use Snakemake?
- What can be done with Snakemake?

Transcript

1. Snakemake in the “Wild” Joshua Friedman January 23rd, 2019
2. Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction 2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion
3. Introduction Concept Snakemake Application Live Demo Conclusion Who am I... • Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes
4. Introduction Concept Snakemake Application Live Demo Conclusion Who am I... • Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes
5. Introduction Concept Snakemake Application Live Demo Conclusion Who am I... • Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes
6. Introduction Concept Snakemake Application Live Demo Conclusion Who am I... • Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes
7. Introduction Concept Snakemake Application Live Demo Conclusion Who am I... • Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes
8. Introduction Concept Snakemake Application Live Demo Conclusion Who am I... • Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes
9. Introduction Concept Snakemake Application Live Demo Conclusion Who am I... • Canadian • Hockey • Mountains • Travel • Maps • Data • QR Codes
10. Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010 2012 2014 2016 2018 2020 BSc Civil
11. Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010 2012 2014 2016 2018 2020 BSc Civil Baird
12. Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010 2012 2014 2016 2018 2020 BSc Civil Baird MSc Coastal
13. Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010 2012 2014 2016 2018 2020 BSc Civil Baird MSc Coastal Deltares
14. Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010 2012 2014 2016 2018 2020 BSc Civil Baird MSc Coastal Deltares TerraLoupe
15. Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010 2012 2014 2016 2018 2020 BSc Civil Baird MSc Coastal Deltares TerraLoupe KI labs
16. Introduction Concept Snakemake Application Live Demo Conclusion Experience 2008 2010 2012 2014 2016 2018 2020 TerraLoupe KI labs
17. Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction 2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion
18. Introduction Concept Snakemake Application Live Demo Conclusion Idea Inspired by a reddit post
19. Introduction Concept Snakemake Application Live Demo Conclusion Idea
20. Introduction Concept Snakemake Application Live Demo Conclusion Idea
21. Introduction Concept Snakemake Application Live Demo Conclusion Idea
22. Introduction Concept Snakemake Application Live Demo Conclusion Idea What about doing this with QR codes?
23. Introduction Concept Snakemake Application Live Demo Conclusion Concept How to execute?
24. Introduction Concept Snakemake Application Live Demo Conclusion Concept How to scale?
25. Introduction Concept Snakemake Application Live Demo Conclusion DAG • Directed Acyclic Graph • Derived from Graph theory • Represents an execution plan • Nodes are jobs • Edges are dependencies DAG paths help differentiate between serial and parallel execution
26. Introduction Concept Snakemake Application Live Demo Conclusion Execution How to define, schedule and execute a DAG? • Workflow Engine • Workflow Manager • Workflow Management System • ... A workflow engine is a software application that manages processes. (Anonymous Wikipedia Contributor)
27. Introduction Concept Snakemake Application Live Demo Conclusion Workflow Managers • Snakemake • NextFlow • Ruffus • Toil • Airflow • Luigi • Pachyderm • Arvados • Taverna • Galaxy • Oozie • Cuneiform • Fireworks • Pypeline • Agave • BioDT • BPipe • Yabi • SciLuigi • Azkaban • Chronos
28. Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction 2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion
29. Introduction Concept Snakemake Application Live Demo Conclusion Overview The Snakemake workflow management system is a tool to create reproducible and scalable data analyses. Workflows are described via a human readable, Python based language. They can be seamlessly scaled to server, cluster, grid and cloud environments, without the need to modify the workflow.
30. Introduction Concept Snakemake Application Live Demo Conclusion Features • Reproducible • Scalable • Implicit DAG Creation • Dynamic Rules • Wildcard Support • Human-readable Code • Remote Storage and Execution • Docker integration • Reporting • Logging • Temporary Files • Dry Run Execution • Parallel Execution • Detailed Run Statistics • Dynamic Resource Contraints • Rule Prioritization
31. Introduction Concept Snakemake Application Live Demo Conclusion Workflow Definition Snakemake workflows are essentially Python scripts extended by declarative code to define rules. Rules describe how to create output files from input files.
32. Introduction Concept Snakemake Application Live Demo Conclusion Workflow Execution Snakemake is primarily a command-line tool, so the snakemake executable is the primary way to execute, debug, and visualize workflows. Default execution with Snakefile in the current directory
33. Introduction Concept Snakemake Application Live Demo Conclusion Workflow Execution Snakemake is primarily a command-line tool, so the snakemake executable is the primary way to execute, debug, and visualize workflows. Dry run to test the workflow
34. Introduction Concept Snakemake Application Live Demo Conclusion Workflow Execution Snakemake is primarily a command-line tool, so the snakemake executable is the primary way to execute, debug, and visualize workflows. Parallel execution based on available cores
35. Introduction Concept Snakemake Application Live Demo Conclusion Workflow Execution Snakemake is primarily a command-line tool, so the snakemake executable is the primary way to execute, debug, and visualize workflows. Visualization of the entire DAG
36. Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction 2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion
37. Introduction Concept Snakemake Application Live Demo Conclusion Setup Python functions with a clear dependency...
38. Introduction Concept Snakemake Application Live Demo Conclusion Workflow - Input String String input for searching GIPHY
39. Introduction Concept Snakemake Application Live Demo Conclusion Workflow - GIPHY JSON output from GIPHY
40. Introduction Concept Snakemake Application Live Demo Conclusion Workflow - QR Codes Embedded GIPHY links
41. Introduction Concept Snakemake Application Live Demo Conclusion Workflow - Input Image Input Face
42. Introduction Concept Snakemake Application Live Demo Conclusion Workflow - Output Output Mosaic
43. Introduction Concept Snakemake Application Live Demo Conclusion Workflow - Output Scan me!
44. Introduction Concept Snakemake Application Live Demo Conclusion Execution DAG for the previous workflow
45. Introduction Concept Snakemake Application Live Demo Conclusion Execution DAG for multiple images
46. Introduction Concept Snakemake Application Live Demo Conclusion Execution DAG for multiple GIPHY search terms
47. Introduction Concept Snakemake Application Live Demo Conclusion Execution DAG for multiple images and multiple GIPHY search terms
48. Introduction Concept Snakemake Application Live Demo Conclusion Sample Output “Silly” Nithish “Silly” Martin
49. Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction 2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion
50. Introduction Concept Snakemake Application Live Demo Conclusion Contents 1. Introduction 2. Concept 3. Snakemake 4. Application 5. Live Demo 6. Conclusion
51. Introduction Concept Snakemake Application Live Demo Conclusion Key Takeaways • Snakemake is an excellent workflow engine • It is flexible, scalable, reproducible, etc... • Snakemake is language and cloud agnostic • QR codes make great gifts and advent calendars
52. Introduction Concept Snakemake Application Live Demo Conclusion Key Takeaways • Snakemake is an excellent workflow engine • It is flexible, scalable, reproducible, etc... • Snakemake is language and cloud agnostic • QR codes make great gifts and advent calendars
53. Introduction Concept Snakemake Application Live Demo Conclusion Key Takeaways • Snakemake is an excellent workflow engine • It is flexible, scalable, reproducible, etc... • Snakemake is language and cloud agnostic • QR codes make great gifts and advent calendars
54. Introduction Concept Snakemake Application Live Demo Conclusion Key Takeaways • Snakemake is an excellent workflow engine • It is flexible, scalable, reproducible, etc... • Snakemake is language and cloud agnostic • QR codes make great gifts and advent calendars
55. Introduction Concept Snakemake Application Live Demo Conclusion Final “Gift”
56. Thank you for your attention. Questions?