Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Data-driven Innovation
Search
Matt Wood
October 10, 2012
Technology
480
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Data-driven Innovation
Slides from my session at the #AWS Public Sector Summit, 2012.
Matt Wood
October 10, 2012
Transcript
123 slides
Data-driven innovation
[email protected]
Dr. Matt Wood @mza
Hello
Hello
Data
DNA
Chromosome 11 : ACTN3 : rs1815739
Chromosome X : rs6625163
Chromosome 19 : FUT2 : rs601338
Chromosome 2 : rs10427255
TYPE II Chromosome 10 : rs7903146
+0.25 Chromosome 15 : rs2472297
I know this, because...
None
A T C G G T C C A G
G
A T C G G T C C A G
G A G C C A G G U C C Transcription
A T C G G T C C A G
G A G C C A G G U C C Translation Ser Glu Val Transcription
None
None
Chromosome 11 : ACTN3 : rs1815739
Chromosome X : rs6625163
Chromosome 19 : FUT2 : rs601338
Chromosome 2 : rs10427255
TYPE II Chromosome 10 : rs7903146
+0.25 Chromosome 15 : rs2472297
I know all that, because...
Human Genome Project
40 species ensembl.org
Compare
Change
Less
None
None
Compare
Transformative
None
Data generation costs are falling everywhere
Customer segmentation, financial modeling, system analysis, line of sight, business
intelligence.
Opportunity
Transformation
Innovation
Generation Collection & storage Analytics & computation Collaboration & sharing
Generation Collection & storage Analytics & computation Collaboration & sharing
lower cost, increased throughput
Generation Collection & storage Analytics & computation Collaboration & sharing
lower cost, increased throughput highly constrained
Barrier
Data generation challenge X
Analytics challenge
Accessibility challenge
Enter the AWS Cloud
Utility
Remove constraints
Data-driven innovation
Distributed
2
2 Software for distributed storage & analysis
2 Software for distributed storage & analysis Infrastructure for distributed
storage & analysis
Software Frameworks for data-intensive work loads. Distributed by design.
Infrastructure Platform for data-intensive work loads. Distributed by design.
Support the data timeline
Generation Collection & storage Analytics & computation Collaboration & sharing
highly constrained
Generation Collection & storage Analytics & computation Collaboration & sharing
Lower the barrier to entry
Agility
Responsive
Generation Collection & storage Analytics & computation Collaboration & sharing
Generation DynamoDB Analytics & computation Collaboration & sharing
Generation DynamoDB EC2, Elastic MapReduce Collaboration & sharing
Generation DynamoDB EC2, Elastic MapReduce S3, Public Datasets
Tools and techniques for working productively with data
Scale
Secure
2 Software for distributed storage & analysis Infrastructure for distributed
storage & analysis
Amazon EC2
Scale out systems Embarrassingly parallel Queue based distribution Small, medium
and high scale
High performance
High performance Compute performance
Cluster Compute Intel Xeon E5-2670 10 gigabit, non-blocking network 60.5
Gb Placement groupings
Cluster Compute Intel Xeon E5-2670 10 gigabit, non-blocking network 60.5
Gb Placement groupings +GPU
240 TFLOPS
High performance Compute performance IO performance
Unstructured
Variable
Amazon DynamoDB Predictable, consistent performance Unlimited storage Single digit millisecond
latencies No schema. Zero admin.
...and SSDs for all
hi1.4xlarge 2 x 1Tb SSD storage 10 gigabit networking HVM:
90k IOPS read, 9k to 75k write PV: 120k IOPS read, 10k to 85k write
Netflix “The hi1.4xlarge configuration is about half the system cost
for the same throughput.” http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html
Provisioned IOPS Provision required IO performance EBS optimized instances
Cost optimization
Reserved capacity
Reserved capacity On-demand
Reserved capacity On-demand
Spot instances
None
$0.2530 vs $2.40
2 Software for distributed storage & analysis Infrastructure for distributed
storage & analysis
map/reduce
Map. Reduce.
Write functions. Scale up.
Hadoop
Undi erentiated heavy lifting
Amazon Elastic MapReduce Managed Hadoop Clusters Easy to provision and
monitor Write two functions. Scale up. Choice of Hadoop flavors
Amazon Elastic MapReduce Integrates with S3 Analytics for DynamoDB Perfect
for Spot pricing
Input data S3
Elastic MapReduce Code Input data S3
Elastic MapReduce Code Name node Input data S3
Elastic MapReduce Code Name node Input data S3 Elastic cluster
Elastic MapReduce Code Name node Input data S3 Elastic cluster
HDFS
Elastic MapReduce Code Name node Input data S3 Elastic cluster
HDFS Queries + BI Via JDBC, Pig, Hive
Elastic MapReduce Code Name node Output S3 + SimpleDB Input
data S3 Elastic cluster HDFS Queries + BI Via JDBC, Pig, Hive
Output S3 + SimpleDB Input data S3
CDC Centers for Disease Control and Prevention
“BioSense 2.0 protects the health of the American people by
providing timely insight into the health of communities, regions, and the nation by o ering a variety of features to improve data collection, standardization, storage, analysis, and collaboration”
Health data Collection & storage Analytics & computation Collaboration &
sharing
Health data Collection & storage Analytics & computation Collaboration &
sharing highly constrained
HIPAA, HITECH, FISMA Moderate
GovCloud
Beyond a definition of Big Data
Chromosome 11 : ACTN3 : rs1815739
Chromosome X : rs6625163
Chromosome 19 : FUT2 : rs601338
Chromosome 2 : rs10427255
TYPE II Chromosome 10 : rs7903146
+0.25 Chromosome 15 : rs2472297
Thank you aws.amazon.com @mza
[email protected]
More Decks by Matt Wood
See All by Matt Wood
Field Notes from Expeditions in the Cloud
mza
2
530
A Platform for Big Data
mza
6
880
The Data Lifecycle
mza
5
620
Provision Throughput Like a Boss
mza
0
550
Impact of Cloud Computing: Life Sciences
mza
2
960
Latency's Worst Nightmare: Performance Tuning Tips and Tricks
mza
4
1.2k
Under the Covers of DynamoDB
mza
4
1.3k
From Analytics to Intelligence: Amazon Redshift
mza
9
1.1k
Scaling Science
mza
3
610
Other Decks in Technology
See All in Technology
MCPでネットワーク障害調査を自動化する前に直すべき3つのこと
yama3133
1
640
その修正で本当に速くなったのか? k6とOpenTelemetryで確かめる性能改善 / Performance review with k6 and OpenTelemetry
ymotongpoo
3
310
あの日、見つかっていなかったらどうなっていたか
kazzpapa3
2
260
落とし穴にハマってわかった 開発生産性を測るときに気をつけたいこと
munakata
3
450
現地速報! Microsoft Ignite アップデートレポート
kasada
0
250
2026-11-16 JAWS-UG コンテナ支部 #31 ゆるLT大会 ECR のライフサイクルポリシーを いい感じに設定したい / ECR Lifecycle policy
masasuzu
3
570
AWS Client VPNの設定でハマったポイントとその過程での気づき
amarelo_n24
1
410
人の努力に頼らないセキュリティをエンジニアリングする在り方 / An approach to engineering security that does not rely on human effort
carta_engineering
0
220
Why start a new company, on Rails and AI, today. SF Ruby Conf 2026, Irina Nazarova
irinanazarova
0
180
SORACOM × AWS IoT Core でSREする
butadora
0
120
Re:ゼロから始めるシェアハウスのペットな鯖
silmin_
0
260
マネージャーをエスパーにしない マネジメントされる側から始める、働きやすい現場づくり
comi190327
1
810
Featured
See All Featured
SERP Conf. Vienna - Web Accessibility: Optimizing for Inclusivity and SEO
sarafernandez
2
1.7k
The Cult of Friendly URLs
andyhume
79
7.1k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4.6k
The Hidden Cost of Media on the Web [PixelPalooza 2025]
tammyeverts
2
580
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
18k
Art, The Web, and Tiny UX
lynnandtonic
304
22k
What the history of the web can teach us about the future of AI
inesmontani
PRO
3
820
Primal Persuasion: How to Engage the Brain for Learning That Lasts
tmiket
0
520
Navigating Algorithm Shifts & AI Overviews - #SMXNext
aleyda
1
1.7k
Making the Leap to Tech Lead
cromwellryan
135
10k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3.6k
The agentic SEO stack - context over prompts
schlessera
0
1k