Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Big Data Analytics
Search
Matt Wood
August 01, 2012
Technology
1.3k
7
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Big Data Analytics
An introduction to Big Data Analytics in the cloud.
Matt Wood
August 01, 2012
Transcript
104 slides
Big Data Analytics w i t h A m a
z o n W e b S e r v i c e s Dr. Matt Wood An Online Seminar for Partners. Wednesday 1st August.
Hello, and thank you.
Big Data Analytics An introduction
Big Data Analytics An introduction The story of analytics on
AWS
Big Data Analytics An introduction The story of analytics on
AWS Integrating partners
Big Data Analytics An introduction The story of analytics on
AWS Integrating partners Partner success stories
INTRODUCING BIG DATA 1
Data for competitive advantage.
Customer segmentation, financial modeling, system analysis, line-of-sight, business intelligence. Using
data
Generation Collection & storage Analytics & computation Collaboration & sharing
Cost of data generation is falling.
Generation Collection & storage Analytics & computation Collaboration & sharing
lower cost, increased throughput
Generation Collection & storage Analytics & computation Collaboration & sharing
HIGHLY CONSTRAINED
Very high barrier to turning data into information.
Move from a data generation challenge to analytics challenge.
Enter the Cloud.
Remove the constraints.
Enable data-driven innovation.
Move to a distributed data approach.
Maturation of two things.
Maturation of two things. Software for distributed storage and analysis
Maturation of two things. Software for distributed storage and analysis
Infrastructure for distributed storage and analysis
Frameworks for data-intensive workloads. Software Distributed by design.
Platform for data-intensive workloads. Infrastructure Distributed by design.
Support the data timeline.
Generation Collection & storage Analytics & computation Collaboration & sharing
HIGHLY CONSTRAINED
Generation Collection & storage Analytics & computation Collaboration & sharing
Lower the barrier to entry.
Accelerate time to market and increase agility.
Enable new business opportunities.
Washington Post Pinterest NASA
“AWS enables Pfizer to explore difficult or deep scientific questions
in a timely, scalable manner and helps us make better decisions more quickly” Michael Miller, Pfizer
THE STORY OF ANALYTICS 2
EC2 Utility computing. 6 years young.
Embarrassingly parallel problems. Scale out systems Queue based distribution. Small,
medium and high scale.
None
None
None
EC2 Utility computing. 6 years young. Cost optimization.
Achieving economies of scale 100% Time
Reserved capacity Achieving economies of scale 100% Time
Reserved capacity Achieving economies of scale 100% Time On-demand
Reserved capacity Achieving economies of scale 100% Time On-demand UNUSED
CAPACITY
Bid on unused EC2 capacity. Spot Instances Very large discount.
Perfect for batch runs. Balance cost and scale.
$650 per hour
Pattern for distributed computing. Map/reduce Software frameworks such as Hadoop.
Write two functions. Scale up.
Pattern for distributed computing. Map/reduce Software frameworks such as Hadoop.
Write two functions. Scale up. Complex cluster configuration and management.
Managed Hadoop clusters. Amazon Elastic MapReduce Easy to provision and
monitor. Write two functions. Scale up. Optimized for S3 access.
Input data S3 UNDER THE HOOD i i
Elastic MapReduce Code Input data S3 UNDER THE HOOD i
i
Elastic MapReduce Code Name node Input data S3 UNDER THE
HOOD i i
Elastic MapReduce Code Name node Input data S3 Elastic cluster
UNDER THE HOOD i i
Elastic MapReduce Code Name node Input data S3 Elastic cluster
HDFS UNDER THE HOOD i i
Elastic MapReduce Code Name node Input data S3 Elastic cluster
HDFS Queries + BI Via JDBC, Pig, Hive UNDER THE HOOD i i
Elastic MapReduce Code Name node Output S3 + SimpleDB Input
data S3 Elastic cluster HDFS Queries + BI Via JDBC, Pig, Hive UNDER THE HOOD i i
Output S3 + SimpleDB Input data S3 UNDER THE HOOD
i i
None
None
None
None
None
None
None
None
None
None
None
None
None
None
Performance
Performance Compute performance
Intel Xeon E5-2670 Cluster Compute 10 gig E non-blocking network
Placement groupings 60.5 Gb UNDER THE HOOD i i
Intel Xeon E5-2670 Cluster Compute 10 gig E non-blocking network
Placement groupings 60.5 Gb UNDER THE HOOD i i + GPU enabled instances
Performance Compute performance
Performance Compute performance IO performance
NoSQL Unstructured data storage.
Predictable, consistent performance DynamoDB Unlimited storage No schema for unstructured
data Single digit millisecond latencies Backed on solid state drives
...and SSDs for all. New Hi1 storage instances.
2 x 1Tb SSDs hi1.4xlarge 10 GigE network HVM: 90k
IOPS read, 9k to 75k write PV: 120k IOPS read, 10k to 85k write UNDER THE HOOD i i
Netflix “The hi1.4xlarge configuration is about half the system cost
for the same throughput.” http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html
EBS Elastic Block Store
Provisioned IOPS Provision required IO performance
Provisioned IOPS Provision required IO performance + EBS-optimized instances with
dedicated throughput
Generation Collection & storage Analytics & computation Collaboration & sharing
Performance + ease of use
PARTNER INTEGRATION 3
Extend platform with partners
Innovate on behalf of customers
Remove undifferentiated heavy lifting
Rolled the Amazon Hadoop optimizations into MapR MapR distribution for
EMR Choice for EMR customers Easy deployment for MapR customers
Hadoop distribution MapR distribution for EMR Integrated into EMR NFS
and ODBC drivers High availability and cluster mirroring
Enterprise data toolchain Informatica on EMR “Swiss army knife” for
data formats Data integration Available to all on EMR
AWS Marketplace Karmasphere, Marketshare, Acunu Cassandra, Metamarkets, Aspera and more.
aws.amazon.com/marketplace
PARTNER SUCCESS STORIES 4
Razorfish
3.5 billion records 71MM unique cookies 1.7MM targeted ads per
day
3.5 billion records 71MM unique cookies 1.7MM targeted ads per
day 500% improvement in return on ad spend.
Cycle Computing + Schrodinger
30k cores, $4200 an hour (compared to $10+ million)
Marketshare + Ticketmaster Optimize live event pricing
Reduced developer infrastructure management time by 3 hours a day
Thank you!
Q & A
[email protected]
@mza on Twitter
More Decks by Matt Wood
See All by Matt Wood
Field Notes from Expeditions in the Cloud
mza
2
530
A Platform for Big Data
mza
6
880
The Data Lifecycle
mza
5
620
Provision Throughput Like a Boss
mza
0
550
Impact of Cloud Computing: Life Sciences
mza
2
960
Latency's Worst Nightmare: Performance Tuning Tips and Tricks
mza
4
1.2k
Under the Covers of DynamoDB
mza
4
1.3k
From Analytics to Intelligence: Amazon Redshift
mza
9
1.1k
Scaling Science
mza
3
610
Other Decks in Technology
See All in Technology
パスワードのない鍵で、 データを手元に置く
ginjihayashi
0
380
業務の傍らで進める改善ループエンジニアリング
bengo4com
1
210
3文字略語に騙されない! 世界一わかりやすい『EDR』超入門
kamadamakoto
1
280
常時稼働エージェントの作り方(Claude公式ブログを資料化)
minorun365
PRO
17
28k
20261113_JAWS_Osaka
takuyay0ne
1
590
Amazon Bedrock AgentCore の基礎とセキュリティ・ガバナンス
ren8k
0
190
速くなるほど、深く考える AI時代のQAに学ぶ
kobayashiyorimitsu
0
270
ぼくのClaude Codeの使い方
falken
PRO
5
7.6k
専門家に使ってもらうAIプロダクトをつくる - 病院の医事課向けDPCコーディング支援システムの設計と評価
hagino3000
4
920
メール到達性を支える、リンク先URLのドメイン評価の扱い方
wind111
0
170
異夢同船〜異なる夢でも同じ船でなんとかする〜
fortegp05
1
550
信頼性を犠牲にせず、PR マージからデプロイまで人の介入を完全にゼロにするまでの道のり
knwoop
2
1.1k
Featured
See All Featured
Crafting Experiences
bethany
1
450
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
uxyall
2
4k
Evolving SEO for Evolving Search Engines
ryanjones
0
370
Chasing Engaging Ingredients in Design
codingconduct
0
400
Why Our Code Smells
bkeepers
PRO
340
58k
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
inesmontani
PRO
0
810
Claude Code どこまでも/ Claude Code Everywhere
nwiizo
68
61k
Are puppies a ranking factor?
jonoalderson
2
4.2k
[Rails World 2026] Durable orchestration on Rails: from continuation to workflow
palkan
4
1.3k
The Illustrated Guide to Node.js - THAT Conference 2024
reverentgeek
1
610
How to Grow Your eCommerce with AI & Automation
katarinadahlin
PRO
2
350
16th Malabo Montpellier Forum Presentation
akademiya2063
PRO
0
470