Slide 1

Slide 1 text

A Platform for Deepak Singh P r i n c i p a l P r o d u c t M a n a g e r Data Science

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

bioinformatics

Slide 4

Slide 4 text

collection

Slide 5

Slide 5 text

curation

Slide 6

Slide 6 text

analysis

Slide 7

Slide 7 text

so what?

Slide 8

Slide 8 text

Image: Yael Fitzpatrick (AAAS)

Slide 9

Slide 9 text

lots of data

Slide 10

Slide 10 text

lots of people

Slide 11

Slide 11 text

lots of places

Slide 12

Slide 12 text

lots of data

Slide 13

Slide 13 text

lots of data really fast

Slide 14

Slide 14 text

constant change

Slide 15

Slide 15 text

we want to make our data more effective

Slide 16

Slide 16 text

versioning

Slide 17

Slide 17 text

provenance

Slide 18

Slide 18 text

filter Via asklar under a CC-BY license

Slide 19

Slide 19 text

aggregate Image: Chris Heiler

Slide 20

Slide 20 text

extend Image: Bethan

Slide 21

Slide 21 text

human interfaces Image: Sebastian Anthony

Slide 22

Slide 22 text

share

Slide 23

Slide 23 text

image: Leo Reynolds communicate

Slide 24

Slide 24 text

hard problem

Slide 25

Slide 25 text

really hard problem

Slide 26

Slide 26 text

so how do you get there?

Slide 27

Slide 27 text

information platforms

Slide 28

Slide 28 text

Image: Drew Conway

Slide 29

Slide 29 text

dataspaces Further reading: Jeff Hammerbacher, Information Platforms and the rise of the data scientist, Beautiful Data

Slide 30

Slide 30 text

the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent Systems, 24, 8-12 (2009)

Slide 31

Slide 31 text

accept all data formats

Slide 32

Slide 32 text

evolve APIs

Slide 33

Slide 33 text

data as a programmable resource

Slide 34

Slide 34 text

Heroku Data Clips

Slide 35

Slide 35 text

Data Clips allow the results of SQL queries on a Heroku Postgres database to be easily shared. Simply create a query on postgres.heroku.com, and then share the resulting URL with co- workers, colleagues, or the world. Data clips can be shared through e-mail, Twitter, irc, or any other medium - they are just URLs. The recipients of a data clip are able to view the data in their browser or download it in JSON, CSV, XML, or Microsoft Excel formats.

Slide 36

Slide 36 text

https://postgres.heroku.com/dataclips/psywmdixgtrxkorpzcueiohkwona

Slide 37

Slide 37 text

data is a royal garden

Slide 38

Slide 38 text

compute is a fungible commodity

Slide 39

Slide 39 text

constraints everywhere

Slide 40

Slide 40 text

Hardware CPU, storage, memory Data management Collections, datasets, provenance Software parallelization, optimization Availability Backup, redundant, replicated Cost Small

Slide 41

Slide 41 text

remove constraints Credit: Pieter Musterd a CC-BY-NC-ND license

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

your infrastructure

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

ec2-run-instances

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

secure global on demand

Slide 49

Slide 49 text

programmable

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

CloudFormation Templates "ClusterUserKeys" : { "Type" : "AWS::IAM::AccessKey", "Properties" : { "UserName" : { "Ref": "ClusterUser" } } }, ! "Ec2Instance" : { ! "Type" : "AWS::EC2::Instance", ! "Properties" : { ! "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ], ! ! ! "InstanceType" : "t1.micro", ! "ImageId" : "ami-7341831a", ! ! "KeyName" : { "Ref" : "KeyName" }, ! "Tags" : [{ ! "Key" : "Role", ! "Value" : "Controller" ! }], ! ! ! "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", ! ! ! [ ! ! ! ! ! ! "#!/bin/sh\n", ! ! ! "/opt/aws/bin/cfn-init ", " -s ", { "Ref" : "AWS::StackName" }, " -r Ec2Instance ", ! ! ! "--access-key=", { "Ref" : "ClusterUserKeys" }, " ", "--secret-key=", { "Fn::GetAtt" : ["ClusterUserKeys", "SecretAccessKey"]}, "\n", ! ! ! ! ! ! "cd /usr/src/pycrypto/pycrypto-2.4; /usr/bin/python setup.py build\n", ! ! ! ! ! ! "cd /usr/src/pycrypto/pycrypto-2.4; /usr/bin/python setup.py install\n", ! ! ! ! ! ! "cd /home/ec2-user/starcluster; /usr/bin/python distribute_setup.py\n", ! ! ! ! ! ! "cd /home/ec2-user/starcluster; /usr/bin/python setup.py install\n", ! ! ! ! ! "/bin/mkdir /home/ec2-user/.starcluster\n", ! ! ! ! ! "/bin/chown ec2-user:ec2-user -R /home/ec2-user/.starcluster\n", ! ! ! ! ! "/usr/bin/ruby /home/ec2-user/parser.rb /home/ec2-user/cc2-template.erb /home/ec2-user/values.yml > /home/ec2- user/.starcluster/config\n", ! ! ! ! ! "/usr/bin/starcluster -c /home/ec2-user/.starcluster/config createkey ", { "Ref" : "ClusterKeypair" }, " -o / home/ec2-user/.ssh/rsa-", { "Ref" : "ClusterKeypair" }, "\n", ! ! ! ! ! "/bin/chown ec2-user:ec2-user -R /home/ec2-user/.ssh/rsa-", { "Ref" : "ClusterKeypair" }, "\n", ! ! ! ! ! "cd /home/ec2-user/; /usr/bin/starcluster -c /home/ec2-user/.starcluster/config start ec2-cluster \n"! ! ! ! ! ! ! ! ! ! ! ! ]]}}

Slide 52

Slide 52 text

elastic

Slide 53

Slide 53 text

Netflix needed to transcode 17,000 titles (80TB of data) to support the launch of Sony PS3. They provisioned 1200 Amazon EC2 instances and completed the transcoding process in just days. Source: Adrian Cockroft (Netflix)

Slide 54

Slide 54 text

Source: Adrian Cockroft (Netflix)

Slide 55

Slide 55 text

scalable

Slide 56

Slide 56 text

Storage: Scale of Amazon S3 Total  Number  of  Objects  Stored  in  Amazon  S3 Q4 2006 Q4 2007 Q4 2008 Q4 2009 Q4 2010 Q4 2011 Peak  Requests: 500,000+ per  second 2.9  Billion 14  Billion 40  Billion 102  Billion 762  Billion 262  Billion

Slide 57

Slide 57 text

durable

Slide 58

Slide 58 text

99.999999999%

Slide 59

Slide 59 text

I did say data was a royal garden

Slide 60

Slide 60 text

performance

Slide 61

Slide 61 text

“Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1 billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets

Slide 62

Slide 62 text

Druid: 26 billion rows/second

Slide 63

Slide 63 text

200 Elastic MapReduce Clusters per day Asynchronous personalization requirements for 39M unique users 300 TB of data per day

Slide 64

Slide 64 text

Cores 17024 Rmax 240.09 Rpeak 354.12 Linpack

Slide 65

Slide 65 text

November 2011

Slide 66

Slide 66 text

“Our tests have shown more than 90 percent scaling efficiency on clusters of up to 128 GPUs each”

Slide 67

Slide 67 text

consumption models

Slide 68

Slide 68 text

on-demand

Slide 69

Slide 69 text

Reserved Instances Light Utilization Medium Utilization Heavy Utilization

Slide 70

Slide 70 text

$0# $5,000# $10,000# $15,000# $20,000# $25,000# 5%# 10%# 15%# 20%# 25%# 30%# 35%# 40%# 45%# 50%# 55%# 60%# 65%# 70%# 75%# 80%# 85%# 90%# 95%# 100%# On#Demand# Light#U

Slide 71

Slide 71 text

what is the value of your data?

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

30,472 cores

Slide 75

Slide 75 text

$1279/hr

Slide 76

Slide 76 text

the clouds biggest value

Slide 77

Slide 77 text

remove constraints

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

Image: Chris Dagdigian

Slide 80

Slide 80 text

http://www.bio-itworld.com/2012/02/14/two-guys-credit-card-metrum-amazon-cloud-makeover.html

Slide 81

Slide 81 text

“Now clients don’t have to wait in a queue… All projects get priority.” -- Adam Kraut

Slide 82

Slide 82 text

queues suck!!!

Slide 83

Slide 83 text

AWS for life sciences

Slide 84

Slide 84 text

services Amazon Elastic Compute Cloud (EC2) Amazon Elastic MapReduce Amazon Simple Storage Service AWS Import/Export Amazon Elastic Block Store Amazon RDS Amazon DynamoDB Amazon VPC AWS CloudFormation

Slide 85

Slide 85 text

services Zero  Administra.on Low  Latency  SSD’s Reserved  Capacity Unlimited  Poten.al  Storage   and  Throughput DynamoDB is a fully managed NoSQL database service that provides extremely fast and predictable performance with seamless scalability

Slide 86

Slide 86 text

Public Data Sets

Slide 87

Slide 87 text

AWS$Public$Data$ EBS$Snapshot$ S3$Bucket$ EC2$Instance$$ (Orchestrator)$ $ $ $ $ $ $ $ CloudForma=on$ Template$ Route 53 Zone Apex (optional) Chef$Cookbook$ Public$AMIs$ Autoscale (optional) Web$Server$ Chef$ Pre/Post$ processing$ End User Developer Console API/ CLI HTTP/HTTPS Client

Slide 88

Slide 88 text

EC2$Instance$$ (Orchestrator)$ $ $ $ $ $ $ $ Route 53 Zone Apex (optional) Autoscale (optional) Web$Server$ Chef$ Pre/Post$ processing$ End User HTTP/HTTPS Client EC2$ $ EC2$ $ EC2$ $ EC2$ $ EC2$ $ Star Cluster Blastz

Slide 89

Slide 89 text

Community http://cloudbiolinux.org/ http://clovr.org/ http://sourceforge.net/projects/seqware

Slide 90

Slide 90 text

Community http://bowtie-bio.sourceforge.net/myrna/index.shtml http://bowtie-bio.sourceforge.net/crossbow/index.shtml

Slide 91

Slide 91 text

Community http://sagebase.org/research/Synapse1.php http://www.genomespace.org/

Slide 92

Slide 92 text

Community http://galaxy.psu.edu/

Slide 93

Slide 93 text

Commercial

Slide 94

Slide 94 text

Getting Stuff Done

Slide 95

Slide 95 text

AMBER Contact Me

Slide 96

Slide 96 text

in summary

Slide 97

Slide 97 text

large scale data requires a rethink

Slide 98

Slide 98 text

data architecture

Slide 99

Slide 99 text

compute architecture

Slide 100

Slide 100 text

distributed, programmable infrastructure

Slide 101

Slide 101 text

No content

Slide 102

Slide 102 text

remove constraints

Slide 103

Slide 103 text

can we build data science platforms?

Slide 104

Slide 104 text

there is no magic there is only awesome

Slide 105

Slide 105 text

[email protected] Twitter:@mndoci http://speakerdeck.com/u/mndoci http://deepaksingh.net Inspiration and ideas from Matt Wood& Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license