Improving Bioinformatic Methods via Enhancing and Scaling Up Workflows

Slide 1

Slide 1 text

Sandra Gesing Center for Research Compu6ng [email protected] 16 March 2016 Improving Bioinforma6c Methods via Enhancing and Scaling Up Workﬂows

Slide 2

Slide 2 text

University of Notre Dame Sandra Gesing 2 hKp://chartsbin.com/view/1124 hKp://chartsbin.com/view/1124 •  In the middle of nowhere of northern Indiana (1.5 h from Chicago) •  4 undergraduate colleges •  ~35 research ins6tutes and centers •  ~12,000 students

Slide 3

Slide 3 text

University of Notre Dame Sandra Gesing 3

Slide 4

Slide 4 text

Center for Research Compu6ng Sandra Gesing 4 •  SoVware development and proﬁling •  Cyberinfrastructure/science gateway development •  Geographical Informa6on Systems •  Visualiza6on Support •  Computa6onal Scien6st support •  Collabora6ve research/ grant development •  System administra6on/ design and acquisi6on •  ~40 researchers, research programmers, HPC specialists CRC and OIT building hKp://crc.nd.edu

Slide 5

Slide 5 text

Center for Research Compu6ng Sandra Gesing 5 •  Computa6onal resources: 25,000 cores+ •  Storage resources: 3 PB •  Visualiza6on systems •  Systems for virtual hos6ng •  Prototype architectures e.g., Docker, OpenStack •  Access and interface to •  XSEDE •  Open Science Grid •  Blue Waters CRC HPC Center (old Union Sta6on)

Slide 6

Slide 6 text

Bioinforma6cs Sandra Gesing 6 •  Genomics •  Proteomics •  Metabolomics •  Immunomics •  System biology •  Molecular simula6ons •  Docking •  Epidemiology •  … Black Swallowtail – larvae and buKerﬂy

Slide 7

Slide 7 text

The Genomics Boom Sandra Gesing 7 February 16, 2001 biotech company Celera February 15, 2001 The Human Genome Project

Slide 8

Slide 8 text

The Genomics Boom Sandra Gesing 8 Craig Venter (leV) and Francis Collins (right)

Slide 9

Slide 9 text

Big Data Sandra Gesing 9 •  Explosion in the quan6ty, variety and complexity of data •  Ques6ons can be answered impossible to even ask about 10 years ago •  Costs far reduced (e.g., Human Genome project, 15 years, ~$2 billion; today ~3 days, $1000)

Slide 10

Slide 10 text

Big Data Sandra Gesing 10 hKp://www.genome.gov/images/content/cost_per_genome_oct2015.jpg

Slide 11

Slide 11 text

Analysis of data Sandra Gesing 11 12181 acatttctac caacagtgga tgaggttgtt ggtctatgtt ctcaccaaat ttggtgttgt 12241 cagtctttta aattttaacc tttagagaag agtcatacag tcaatagcct tttttagctt 12301 gaccatccta atagatacac agtggtgtct cactgtgatt ttaatttgca ttttcctgct 12361 gactaattat gttgagcttg ttaccattta gacaacttca ttagagaagt gtctaatatt 12421 taggtgactt gcctgttttt ttttaattgg gatcttaatt tttttaaatt attgatttgt 12481 aggagctatt tatatattct ggatacaagt tctttatcag atacacagtt tgtgactatt 12541 ttcttataag tctgtggttt ttatattaat gtttttattg atgactgttt tttacaattg 12601 tggttaagta tacatgacat aaaacggatt atcttaacca ttttaaaatg taaaattcga 12661 tggcattaag tacatccaca atattgtgca actatcacca ctatcatact ccaaaagggc 12721 atccaatacc cattaagctg tcactcccca atctcccatt ttcccacccc tgacaatcaa 12781 taacccattt tctgtctcta tggatttgcc tgttctggat attcatatta atagaatcaa Slide copied from: Stuart Owen „Workﬂows with Taverna“ A sequence of connected steps in a deﬁned order based on their control and data dependencies

Slide 12

Slide 12 text

Analysis of data Sandra Gesing 12 12181 acatttctac caacagtgga tgaggttgtt ggtctatgtt ctcaccaaat ttggtgttgt 12241 cagtctttta aattttaacc tttagagaag agtcatacag tcaatagcct tttttagctt 12301 gaccatccta atagatacac agtggtgtct cactgtgatt ttaatttgca ttttcctgct 12361 gactaattat gttgagcttg ttaccattta gacaacttca ttagagaagt gtctaatatt 12421 taggtgactt gcctgttttt ttttaattgg gatcttaatt tttttaaatt attgatttgt 12481 aggagctatt tatatattct ggatacaagt tctttatcag atacacagtt tgtgactatt 12541 ttcttataag tctgtggttt ttatattaat gtttttattg atgactgttt tttacaattg 12601 tggttaagta tacatgacat aaaacggatt atcttaacca ttttaaaatg taaaattcga 12661 tggcattaag tacatccaca atattgtgca actatcacca ctatcatact ccaaaagggc 12721 atccaatacc cattaagctg tcactcccca atctcccatt ttcccacccc tgacaatcaa 12781 taacccattt tctgtctcta tggatttgcc tgttctggat attcatatta atagaatcaa Slide copied from: Stuart Owen „Workflows with Taverna“ A sequence of connected steps in a defined order based on their control and data dependencies Workflows!

Slide 13

Slide 13 text

Workflows Sandra Gesing 13 •  Different workflow concepts •  Different workflow languages •  Different workflow constructs Taverna

Slide 14

Slide 14 text

Workflow Editors Sandra Gesing 14 •  Different technologies (workbenches, web-‐based) •  Different look-‐and-‐feel

Slide 15

Slide 15 text

State of the Art Sandra Gesing 15 Data and compute-‐ intensive problems High-‐speed networks Users generally not IT specialists Tools and workﬂow engines Web-‐based agile frameworks Distributed data and compu6ng infrastructures

Slide 16

Slide 16 text

Challenge for Developers Sandra Gesing 16 Data and compute-‐ intensive problems High-‐speed networks Tools and workflow engines Web-‐based agile frameworks Distributed data and compu6ng infrastructures Users generally not IT specialists Need for intui6ve and efficient workflows!

Slide 17

Slide 17 text

Challenge for Developers Sandra Gesing 17 Data and compute-‐ intensive problems High-‐speed networks Tools and workﬂow engines Web-‐based agile frameworks Distributed data and compu6ng infrastructures Users generally not IT specialists

Slide 18

Slide 18 text

Challenge for Developers Sandra Gesing 18

Slide 19

Slide 19 text

Usability Sandra Gesing 19 “AVer all, usability really just means that making sure that something works well: that a person … can use the thing -‐ whether it's a Web site, a ﬁghter jet, or a revolving door -‐ for its intended purpose without geung hopelessly frustrated.” (Steve Krug in “Don't make me think!: A Common Sense Approach to Web Usability”, 2005)

Slide 20

Slide 20 text

Usability Sandra Gesing 20 ... and users should get more features easily...

Slide 21

Slide 21 text

Workflow Enhancements Sandra Gesing 21 •  Logical level: Meta-‐workflows Herres-‐Pawlis, S., Hoffmann, A., Rösener, T., Krüger, J., Grunzke, R., and Gesing, S. “Mul6-‐layer Meta-‐ metaworkflows for the Evalua6on of Solvent and Dispersion Effects in Transi6on Metal Systems Using the MoSGrid Science Gateways”Science Gateways (IWSG), 2015 7th Interna6onal Workshop on, pp.47-‐52, 3-‐5 June 2015, IEEE Xplore, doi: 10.1109/IWSG.2015.13 •  System level: Combina6on of strengths of workflow systems Hazekamp, N., Sarro, J., Choudhury, O., Gesing, S., ScoK Emrich and Thain, D. “Scaling Up Bioinforma6cs Workflows with Dynamic Job Expansion: A Case Study Using Galaxy and Makeflow”, e-‐Science (e-‐Science), 2015 IEEE 11th Interna6onal Conference on, pp.332-‐341, Aug. 31 2015-‐Sept. 4 2015 •  Predic6on: Model for op6miza6on of tasks and threads Choudhury, O., Rajan, D., Hazekamp, N., Gesing, S., Thain, D., and Emrich, S. “Balancing Thread-‐level and Task-‐ level Parallelism for Data-‐Intensive Workloads on Clusters and Clouds”, Cluster Compu6ng (CLUSTER), 2015 IEEE Interna6onal Conference on, pp.390-‐393, 8-‐11 Sept. 2015, doi:10.1109/CLUSTER.2015.60

Slide 22

Slide 22 text

Molecular Simula6ons Sandra Gesing 22 •  Predic6on and analysis of molecular structures •  Numerous applica6ons, e.g. •  Materials science •  Drug design ligands target docking ?

Slide 23

Slide 23 text

Molecular Simula6ons ligands target docking binding energy scoring func6ons binding pocket •  Predic6on and analysis of molecular structure •  Numerous applica6ons, e.g. •  Materials science •  Drug design Sandra Gesing 23

Slide 24

Slide 24 text

MoSGrid Science Gateway Molecular Simula6on Grid •  Science gateway integrated with underlying compute and data management infrastructure •  Distributed workﬂow management •  Data repository •  Metadata management Sandra Gesing 24

Slide 25

Slide 25 text

MoSGrid Science Gateway User Interface WS-‐PGRADE Liferay DCI Resources Middleware Layer UNICORE XtreemFS High-‐Level Middleware Service Layer gUSE Sandra Gesing 25

Slide 26

Slide 26 text

MoSGrid Science Gateway •  Three domains (QC, MD, Docking) •  Gaussian, NWCHem, Gromacs, CADDSuite, … •  Compute and data intensive •  embarrassingly parallel, MPI and GPUs •  Shared workﬂows, metaworkﬂows, data and knowledge •  Distributed compu6ng and data management via UNICORE and XtreemFS Sandra Gesing 26

Slide 27

Slide 27 text

Quantum Chemistry Sandra Gesing 27

Slide 28

Slide 28 text

Quantum Chemistry Sandra Gesing 28

Slide 29

Slide 29 text

Molecular Dynamics Sandra Gesing 29

Slide 30

Slide 30 text

Docking Sandra Gesing 30

Slide 31

Slide 31 text

Visualiza6on Tes6ng of ChemDoodle and MolCAD web.chemdoodle.com molcad.de Sandra Gesing 31

Slide 32

Slide 32 text

Science Case: Polymerisa6on catalysts Sandra Gesing 32

Slide 33

Slide 33 text

Transla6on into Workﬂows Sandra Gesing 33

Slide 34

Slide 34 text

Transla6on into Workﬂows Sandra Gesing 34

Slide 35

Slide 35 text

Meta-‐Workﬂows Sandra Gesing 35

Slide 36

Slide 36 text

Transla6on into Meta-‐Workﬂows Sandra Gesing 36

Slide 37

Slide 37 text

Scaling Up Workﬂows Sandra Gesing 37 # Machines # Cores Data ParBBoning Save

Slide 38

Slide 38 text

Scaling Up Workﬂows Sandra Gesing 38 Galaxy

Slide 39

Slide 39 text

Scaling Up Workﬂows Sandra Gesing 39 Simple Workﬂow in Galaxy Problem: As Size increases so does Time

Slide 40

Slide 40 text

Scaling Up Workﬂows Sandra Gesing 40 Workﬂow with Parallelism added in Galaxy Problem: Tools must be updated every change in Parallelism/Relies on Scien6st

Slide 41

Slide 41 text

Scaling Up Workﬂows Sandra Gesing 41 Workﬂow Dynamically Expanded behind Galaxy

Slide 42

Slide 42 text

Scaling Up Workﬂows Sandra Gesing 42

Slide 43

Slide 43 text

Scaling Up Workﬂows Sandra Gesing 43

Slide 44

Slide 44 text

Scaling Up Workﬂows Sandra Gesing 44 Makeﬂow •  Task Structure INPUTS : OUTPUTS COMMAND •  Directed Acyclic Graph (DAG) •  Programma6cally Generated

Slide 45

Slide 45 text

Scaling Up Workﬂows Sandra Gesing 45

Slide 46

Slide 46 text

Scaling Up Workﬂows Sandra Gesing 46

Slide 47

Slide 47 text

Scaling Up Workﬂows Sandra Gesing 47 Job Sandbox – Log ﬁle crea6on for cleanup

Slide 48

Slide 48 text

Scaling Up Workﬂows Sandra Gesing 48

Slide 49

Slide 49 text

Scaling Up Workﬂows Sandra Gesing 49 Using Dynamic Job Expansion we were able to scale up a workﬂow without requiring the huge amount of 6me to process it Found viable solu6ons for: •  Using Work Queue we u6lized 100s of cores from a Condor Pool •  Cleaning Sandbox using knowledge of intermediates and logging •  Explored methods to transmit needed environments such as executables and Java 61.5X speed-‐up on 32 GB dataset u6lizing these methods

Slide 50

Slide 50 text

Thread-‐level and Task-‐level Parallelism Sandra Gesing 50 •  Develop predictive performance models for an application domain •  Achieve acceptable performance the first time •  Optimize resource utilization •  Execution time •  Memory usage

Slide 51

Slide 51 text

Thread-‐level and Task-‐level Parallelism •  WorkQueue master-worker framework •  Sun Grid Engine (SGE) batch system Sandra Gesing 51

Slide 52

Slide 52 text

Thread-‐level and Task-‐level Parallelism Sandra Gesing 52 1.  Applica6on-‐level model for 6me: (,,)= 1/ + 2 2.  Applica6on-‐level model for memory: (,)= γ1R +γ2N 3.  System-‐level model for 6me: =1/ +2(/ +/ )+3T(R,/ ,)∗/ + 4/ +5/  4.  System-‐level model for memory: (,)=ϕ1R +ϕ2Q

Slide 53

Slide 53 text

Thread-‐level and Task-‐level Parallelism Sandra Gesing 53 7 data points (R) 7 data points (Q) 7 data points (N) 343 data points Data CollecBon Training data Regression Model Training Accuracy Test MAPE TesBng Regression Coeﬃcient s Tes6n g data

Slide 54

Slide 54 text

Thread-‐level and Task-‐level Parallelism Sandra Gesing 54 Avg. MAPE = 3.1 MAPE = Mean Absolute Percentage Error

Slide 55

Slide 55 text

Thread-‐level and Task-‐level Parallelism Sandra Gesing 55 For the given dataset, K* = 90, N* = 4

Slide 56

Slide 56 text

Result Sandra Gesing 56 # Cores/ Task # Tasks Predicted Time (min) Speedup Estimated EC2 Cost ($) Estimated Azure Cost ($) 1 360 70 6.6 50.4 64.8 2 180 38 12.3 25.2 32.4 4 90 24 19.5 18.9 32.4 8 45 27 17.3 18.9 32.4

Slide 57

Slide 57 text

Challenges – Novel and Old... Sandra Gesing Science Gateways 57 … require novel solu6ons!

Slide 58

Slide 58 text

EU COST Ac6on cHiPSet (IC1406) Sandra Gesing Science Gateways 58 cHiPSet – High Performance Modeling and Simula6on for Big Data Applica6ons •  April 2015 – April 2019 •  15 countries -‐ 12 COST, 3 non-‐COST (US, China, Australia) •  37 reseach organiza6ons/companies (31 COST, 6 non-‐ COST) hKp://www.cost.eu/COST_Ac6ons/ict/Ac6ons/IC1406

Slide 59

Slide 59 text

EU COST Ac6on cHiPSet Sandra Gesing Science Gateways 59

Slide 60

Slide 60 text

cHiPSet -‐ Collabora6ons Sandra Gesing Science Gateways 60 Partner Projects •  NESUS (Network for Sustainable Ultrascale Compu6ng) hKp://www.nesus.eu/ •  MPM4CPS (Mul6-‐Paradigm Modelling for Cyber-‐ Physical Systems) hKp://www.mpm4cps.eu/ And maybe YOU?

Slide 61

Slide 61 text

Acknowledgements Sandra Gesing Science Gateways 61 Logical level •  Richard Grunzke •  Sonja Herres-‐Pawlis •  Alexander Hoﬀmann •  Jens Krüger •  MTA SZTAKI •  University of Westminster System level and predic6on •  Nicholas Hazekamp •  Olivia Choudhury •  Douglas Thain •  Scott Emrich •  Notre Dame Bioinformatics Lab •  The Cooperative Computing Lab, University of Notre Dame

Slide 62

Slide 62 text

Sandra Gesing Science Gateways 62 [email protected]