Building petabyte-scale comparative genomics pipelines

This talk will educate the audience about Python tools and best practices for creating reproducible petabyte-scale pipelines. This is done within the context of demonstrating a new grammar-based approach to comparative genomics. The genome grammars are produced using public data from the National Institutes of Health, streamed over a high-throughput Internet2 connection to Amazon Web Services.

Transcript

1. Chris Cope and Jim DeLeo SciPy 2014 Building petabyte-scale comparative genomics pipelines
2. NOT A BIOLOGIST Disclaimer:
3. Mission Empower citizen scientists to analyze the many petabytes of public genomic data • Cloud • Internet2 • Hide the plumbing (mrjob) and Sequitur algorithm for genomics
4. Biology stuff • Next-gen sequencing • Alignment • File formats ◦ FASTA ◦ BAM / SAM • NIH/NLM/NCBI More details on the internets
5. Genomics, why?
6. Sequence Alignment Map
9. Sequitur algorithm
10. resilientscience.github.io/wot WOT WOT?
11. Comparative genomics Lexical analysis • alignment • “grep” Syntactic analysis • grammar • CFG comparison CTGACTGTCGACCTCACGAAGTCCGCCGTAAGC CTGACACCCGACCTCGCGAAGTCCGCCAAGCTC
12. Computing stuff • AWS (Cloud, MapReduce) • National Library of Medicine • Internet2 • Grammars
13. Mission “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.” - Andrew Tanenbaum
15. NIH - AWS 25 miles, 100 Gbps
16. Mission ftp.ncbi.nlm.nih.gov
17. Mission pip install mrjob
20. Untuned end to end throughput = 3 GiB/s = 180 GiB/min
21. What’s next? • phenotype - genotype cookoff ◦ alignment ◦ HMM ◦ Expert Systems (Jim) • Sequitur for building new, and verifying old Expert Systems • EHR • Imaging • scikit-bio (yay SciPy2014!)
22. Acknowledgements Jim DeLeo NIH (CC) Jonathan Simon FDA Ben Busby NIH (NCBI) Jon Riehl Resilient WOT talk tomorrow 3:30 resilientscience.github.io/wot @chris_cope ccope@resilientscience.com