Talk at West Coast Association of Shared Directors meeting

Transcript

1. Platforms for Deepak Singh P r i n c i p a l P r o d u c t M a n a g e r Data Science
4. bioinformatics
5. collection
6. curation
7. analysis
8. so what?
9. Image: Yael Fitzpatrick (AAAS)
10. lots of data
11. lots of people
12. lots of places
13. constant change
14. we want to make our data more effective
15. versioning
16. provenance
17. filter Via asklar under a CC-BY license
18. aggregate Image: Chris Heiler
19. extend Image: Bethan
20. human interfaces Image: Sebastian Anthony
21. share
22. image: Leo Reynolds communicate
23. hard problem
24. really hard problem
25. so how do get there?
26. information platforms
27. Image: Drew Conway
28. dataspaces Further reading: Jeff Hammerbacher, Information Platforms and the rise of the data scientist, Beautiful Data
29. the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent Systems, 24, 8-12 (2009)
30. accept all data formats
31. evolve APIs
32. data as a programmable resource
33. data is a royal garden
34. compute is a fungible commodity
35. constraints everywhere
36. Hardware CPU, storage, memory Data management Collections, datasets, provenance Software parallelization, optimization Availability Backup, redundant, replicated Cost Small
37. remove constraints Credit: Pieter Musterd a CC-BY-NC-ND license
38. amazon web services
39. your infrastructure
41. ec2-run-instances
43. secure global on demand
44. programmable
48. elastic
49. Netflix needed to transcode 17,000 titles (80TB of data) to support the launch of Sony PS3. They provisioned 1200 Amazon EC2 instances and completed the transcoding process in just days. Source: Adrian Cockroft (Netflix)
50. Source: Adrian Cockroft (Netflix)
51. durable
52. 99.999999999%
53. I did say data was a royal garden
54. performance
55. “Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1 billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
56. WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors) Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
57. “Our tests have shown more than 90 percent scaling efficiency on clusters of up to 128 GPUs each”
58. consumption models
59. on-demand
60. Reserved Instances
61. what is the value of your data?
64. the clouds biggest value
65. remove constraints
67. Image: Chris Dagdigian
68. Credit: Angel Pizzaro, U. Penn
69. 13k sequences - 10 min - 0.1s per sequence
70. mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml
74. 30,472 cores
75. $1279/hr
76. http://cloudbiolinux.org/
77. http://usegalaxy.org/cloud
78. “The process of moving StarMolsim over to the cloud to support the “Introduction to Modeling and Simulation” course at MIT was a huge success. The cloud enabled the STAR group to move away from the responsibility of owning and maintaing dedicated hardware and instead focus on their core mission of developing software and services for faculty, students, and researchers at MIT” http://web.mit.edu/stardev/cluster/about.html
79. in summary
80. large scale data requires a rethink
81. data architecture
82. compute architecture
83. distributed, programmable infrastructure
84. amazon web services
85. remove constraints
86. can we build data science platforms?
87. there is no magic there is only awesome
88. deesingh@amazon.com Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.com Inspiration and ideas from Matt Wood& Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license