Platforms for scientific data analysis

Presented at a Big Data seminar series at Novo Nordisk recently

Transcript

1. There is no magic, only awesome
 Building a platform for scientific data analysis D e e p a k S i n g h ( @ m n d o c i ) P r i n c i p a l P r o d u c t M a n a g e r - A m a z o n E C 2
2. Good morning!
6. bioinformatics
7. Collection
8. Curation
9. Analysis
10. so what?
11. Image: Yael Fitzpatrick (AAAS)
12. lots of data
13. lots of people
14. lots of places
15. constant change
16. we want to make our data more effective
17. versioning
18. provenance
19. filter Via asklar under a CC-BY license
20. Image: Chris Heiler aggregate
21. extend Image: Bethan
22. human interfaces Image: Sebastian Anthony
23. share
24. image: Leo Reynolds communicate
25. hard problem
26. ^ Really hard problem
27. How can we be more effective with data?
28. Wrong question
29. How can we be more effective with our people?
30. text Image: Pieter Musterd Remove constraints
31. Move data to the users?
32. Move tools to the data
33. Give tools access to data
34. Give data access to tools
35. APIs
36. so how do 
 get there?
37. information platforms
38. the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent Systems, 24, 8-12 (2009)
39. accept all data formats
40. evolve APIs
41. data is a programmable resource
42. compute is a 
 fungible commodity
44. Amazon EC2
46. ec2-run-instances
47. programmable
51. Building Blocks
54. Genomic  data  is  assembled  and  analyzed  in  Complete   Genomics’  data  center  and  then  is  securely  transferred  over  a   dedicated  network  to  Amazon  Web  Services  (AWS)  for  delivery   to  customers  either  by  shipping  hard  disk  drives  or   electronically. Automatic  transfer  of  raw  data  to  the  cloud  in  real  time  during  the   course  of  the  sequencing  run   http://blog.basespace.illumina.com/2012/08/10/basespace-­‐growth-­‐the-­‐ numbers/     • 70%  of  all  installed  MiSeqs  have  connected  to  BaseSpace   • BaseSpace  on  HiSeq  in  Q4  2012 Sequencing data going straight to the cloud
55. Analyze data in the cloud
56. Computational compound analysis
 Solar panel material 
 Estimated computation time 264 years
 
 156,314 core cluster across 8 regions 1.21 petaFLOPS (Rpeak) Simulated 205,000 materials 18 hours for $33,000 16¢ per molecule 1 c|net news ! http://news.cnet.com/8301-1001_3-57611919-92/supercomputing-simulation-employs-156000-amazon-processor-cores/
57. Migrated clinical trials simulations platform Simulations in 1.2hrs vs. 60hrs! 64% reduction in costs Clinical Pharmacology & Pharmacometrics Molecular Dynamics Computational Genomics Research Application Portfolio 98% time saved for clinical trial simulations Internal System Cloud Individual Clinical Trial Simulation Run Time (Min) 56 56 Total Number of Clinical Trial Simulations 2000 2000 No. Servers 2 256 No. CPU’s 32 2048 Total Analysis Run Time (hr) 60 1.2 Cost ?? $336
58. CHARGE Consortium
 Aimed at better understanding how human genetics contributes 
 to heart disease and aging
60. Collaborating around data: National Database for Autism Research
61. • An online environment to drive collaboration among researchers • Synapse hosts clinical- genomic datasets • Provides a shared compute space and suite of analysis tools for researchers
62. Global Collaboration for Global Manufacturing Cloud provides a global, distributed, secure, and scalable environment for collaborative design and manufacturing
63. Reproduce. Reuse. Remix
65. https://github.com/awslabs/cfncluster
66. [plugin ipcluster] setup_class = ipcluster.IPCluster enable_notebook = true notebook_passwd = YOUR-PASS ! [cluster qiime]
 node_image_id = ami-2faa7346
 keyname = YOUR-KEY
 cluster_size = 4
 node_instance_type = m2.4xlarge
 plugins = ipcluster ! $ starcluster start -c qiime myqiime Source: Justin Riley
68. 1000 Genomes
69. 1000 Genomes Cloud BioLinux
71. Your HiSeq data Illumina BaseSpace
72. Your data GenomeSpace
74. Reproduce. Reuse. Remix
75. Image: Pieter Musterd Making people more effective
76. Thank You