2019-09-20_-_Botvinnik_-_Nextflow_Camp.pdf

Slide 1

Slide 1 text

Olga Botvinnik, PhD Data Scientist olga.botvinnik@czbiohub.org @olgabot @BioinformaticsBeyonce Nextﬂow at CZ Biohub 2019-09-20 Nextﬂow Camp 2019 +

Slide 2

Slide 2 text

Olga Botvinnik, PhD Data Scientist olga.botvinnik@czbiohub.org @olgabot @BioinformaticsBeyonce Nextﬂow at CZ Biohub 2019-09-20 Nextﬂow Camp 2019 + Defended my PhD in a Quinceañera dress (#phdnera on twitter)

Slide 3

Slide 3 text

Overview What is CZB? NF @ Biohub OUTLINE • Introduction to Chan Zuckerberg Biohub + Myself • What is CZ Biohub? How does it relate to Chan Zuckerberg Initiative? • Computation team, infrastructure, and training at CZ Biohub • Brief overview of my research • Nextflow at CZ Biohub • Other workflow managers we tried • Timeline of Nextflow at Biohub • What debugging a workflow actually looks lik !2

Slide 4

Slide 4 text

Overview What is CZB? NF @ Biohub THE CZ BIOHUB (I) !3 ≠

Slide 5

Slide 5 text

Overview What is CZB? NF @ Biohub THE CZ BIOHUB (I) !3 ≠ Sorry, I can’t help you get a CZI grant …

Slide 6

Slide 6 text

Overview What is CZB? NF @ Biohub THE CZ BIOHUB (II) Independent, 501(c)(3) non-proﬁt, medical research organization Founded in 2016 with a $600 million gift from Priscilla Chan and Mark Zuckerberg Based in San Francisco Satellite facilities on the Stanford and UC Berkley campuses Innovator in the use of technology, engineering and science to accelerate biomedical discoveries and advances Collaborator with UC Berkeley, UCSF and Stanford University – bringing together three powerhouse research institutions in the ﬁght against human disease !4

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Overview What is CZB? NF @ Biohub THE CZ BIOHUB (III) !5

Slide 9

Slide 9 text

Overview What is CZB? NF @ Biohub THE CZ BIOHUB (III) !5 Promote scientiﬁc research and technology development to enable doctors to cure, prevent or manage all diseases during our children’s lifetime.

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Overview What is CZB? NF @ Biohub DATA SCIENCES AND INFORMATION TECHNOLOGY PLATFORM !6 Jim Karkanias Joshua Batson James Webber Aaron McGeever Angela Oliveira Pisco Jenny Folkesson Samantha Hao Phoenix Logan Giana Cirolia Olga Botvinnik Saransh Kaul Lekha Karanam Jack Kamm David Dynerman Lucy Li Pranathi Vemuri Jim Karkanias Saba Nafees Clarissa Vasquez

Slide 12

Slide 12 text

Overview What is CZB? NF @ Biohub COMPUTE INFRASTRUCTURE AT CZ BIOHUB !7 •Biohub datacenter has a significant amount of computation, 3.5PB storage, and 40 GB bandwidth •The largest workloads involve demultiplexing of Novaseq data, single-cell gene expression analysis, and computational microscopy projects. •In AWS, 70 instances are always running which frequently scales up to 170 based on projects. The largest spikes are unsurprisingly around paper completion targets. •AWS currently contains 950TB of data with about 50TB of I/O each month •We utilize about 1.1M compute hours per year in AWS  Jim Karkanias

Slide 13

Slide 13 text

Overview What is CZB? NF @ Biohub INFORMAL WEEKLY TRAINING THROUGH CARBS & COMPUTERS !8

Slide 14

Slide 14 text

Overview What is CZB? NF @ Biohub INFORMAL WEEKLY TRAINING THROUGH CARBS & COMPUTERS !8 Beginner-friendly

Slide 15

Slide 15 text

Overview What is CZB? NF @ Biohub INFORMAL WEEKLY TRAINING THROUGH CARBS & COMPUTERS !8 Beginner-friendly Moved to morning time slot to accommodate time-constrained wet lab experiments

Slide 16

Slide 16 text

Overview What is CZB? NF @ Biohub INFORMAL WEEKLY TRAINING THROUGH CARBS & COMPUTERS !8 Beginner-friendly Intermediate/Advanced Moved to morning time slot to accommodate time-constrained wet lab experiments

Slide 17

Slide 17 text

Overview What is CZB? NF @ Biohub MY RESEARCH (I): COMMON LANGUAGE TO EMBED CELL TYPES ACROSS SPECIES !9 AO Pisco et al, biorxiv (2019) Tabula Muris Senis

Slide 18

Slide 18 text

Overview What is CZB? NF @ Biohub MY RESEARCH (I): COMMON LANGUAGE TO EMBED CELL TYPES ACROSS SPECIES !9 AO Pisco et al, biorxiv (2019) Tabula Muris Senis Neurons Epithelial cells Stem cells

Slide 19

Slide 19 text

Overview What is CZB? NF @ Biohub MY RESEARCH (II): K-MER SIMILARITY OF RNA-SEQ TRANSCRIPTOMES !10 Method: Chop up RNA-seq reads into k-long words (k-mers), re-encode alphabet, hash to integer and compare

Slide 20

Slide 20 text

Overview What is CZB? NF @ Biohub MY RESEARCH (II): K-MER SIMILARITY OF RNA-SEQ TRANSCRIPTOMES !10 -2449632760233112660 599809299905293014 2064419641233326469 475453634193608662 Method: Chop up RNA-seq reads into k-long words (k-mers), re-encode alphabet, hash to integer and compare hash Use hashes for comparison Original: MKKVTAEAISWNESTSETN Dayhoff: eddebbcbebfccbbbcbc HP: hpphphphhphpppppppp Re-encode to a lossy alphabet

Slide 21

Slide 21 text

Overview What is CZB? NF @ Biohub MY RESEARCH (II): K-MER SIMILARITY OF RNA-SEQ TRANSCRIPTOMES !10 Gene expression Nearest neighbor graphs, n_neighbors=5 -2449632760233112660 599809299905293014 2064419641233326469 475453634193608662 Results: Nearest neighbor graphs on mouse bladder, 10x/droplet-based Single-cell RNA seq Method: Chop up RNA-seq reads into k-long words (k-mers), re-encode alphabet, hash to integer and compare hash Use hashes for comparison Original: MKKVTAEAISWNESTSETN Dayhoff: eddebbcbebfccbbbcbc HP: hpphphphhphpppppppp Re-encode to a lossy alphabet

Slide 22

Slide 22 text

Overview What is CZB? NF @ Biohub MY RESEARCH (II): K-MER SIMILARITY OF RNA-SEQ TRANSCRIPTOMES !10 k-mer presence/absence Gene expression Observe 1/1000 k-mers Ksize: 27 Molecule: cDNA Nearest neighbor graphs, n_neighbors=5 -2449632760233112660 599809299905293014 2064419641233326469 475453634193608662 Results: Nearest neighbor graphs on mouse bladder, 10x/droplet-based Single-cell RNA seq Method: Chop up RNA-seq reads into k-long words (k-mers), re-encode alphabet, hash to integer and compare hash Use hashes for comparison Original: MKKVTAEAISWNESTSETN Dayhoff: eddebbcbebfccbbbcbc HP: hpphphphhphpppppppp Re-encode to a lossy alphabet

Slide 23