Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Intel Theater Presentation @ SC11
Search
Deepak Singh
November 19, 2011
Technology
6
180
Intel Theater Presentation @ SC11
Presented at the Intel Theater at SC11
Deepak Singh
November 19, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
100
Platforms for scientific data analysis
mndoci
3
100
FGED Keynote
mndoci
3
92
Open Mic Science - May 7, 2012
mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
250
A Platform for Data Science
mndoci
6
14k
Talk at West Coast Association of Shared Directors meeting
mndoci
3
150
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
110
Platforms for Data Science
mndoci
3
200
Other Decks in Technology
See All in Technology
DevIO2025_継続的なサービス開発のための技術的意思決定のポイント / how-to-tech-decision-makaing-devio2025
nologyance
1
380
バッチ処理で悩むバックエンドエンジニアに捧げるAWS Glue入門
diggymo
3
200
データアナリストからアナリティクスエンジニアになった話
hiyokko_data
2
440
AWSを利用する上で知っておきたい名前解決のはなし(10分版)
nagisa53
10
3.1k
AIエージェント開発用SDKとローカルLLMをLINE Botと組み合わせてみた / LINEを使ったLT大会 #14
you
PRO
0
110
新アイテムをどう使っていくか?みんなであーだこーだ言ってみよう / 20250911-rpi-jam-tokyo
akkiesoft
0
220
ChatGPTとPlantUML/Mermaidによるソフトウェア設計
gowhich501
1
130
RSCの時代にReactとフレームワークの境界を探る
uhyo
10
3.4k
Automating Web Accessibility Testing with AI Agents
maminami373
0
1.2k
JTCにおける内製×スクラム開発への挑戦〜内製化率95%達成の舞台裏/JTC's challenge of in-house development with Scrum
aeonpeople
0
210
Obsidian応用活用術
onikun94
2
480
ブロックテーマ時代における、テーマの CSS について考える Toro_Unit / 2025.09.13 @ Shinshu WordPress Meetup
torounit
0
120
Featured
See All Featured
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.1k
Typedesign – Prime Four
hannesfritz
42
2.8k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Mobile First: as difficult as doing things right
swwweet
224
9.9k
Code Reviewing Like a Champion
maltzj
525
40k
Become a Pro
speakerdeck
PRO
29
5.5k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Navigating Team Friction
lara
189
15k
Fireside Chat
paigeccino
39
3.6k
Transcript
HPC with Amazon EC2 Deepak Singh @mndoci P r i
n c i p a l P r o d u c t M a n a g e r
Amazon Web Services
4
2
1. Infrastructure
None
ec2-run-instances
None
secure global on demand
programmable
None
None
None
elastic
None
instance types
standard (m1) high memory (m2) high CPU (c1) t1.micro
high performance
“Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1
billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
cluster computing
MPI
bandwidth intensive
Cluster Compute Instance
2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7
TB disk HVM cc1.4xlarge
10 gig E
Placement Group
Placement group full- bisection
linpack
Cores 7040 Rmax 41.82 Rpeak 82.51
231 November 2010
451 June 2011
WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)
Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
New Cluster Compute Instance
2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB
disk HVM cc2.8xlarge
linpack
Cores 17024 Rmax 240.09 Rpeak 354.12
42 November 2011
optimizing costs
on-demand
reserved
spot
None
None
None
30,472 cores
$1279/hr
2. Orchestration
None
AWS CloudFormation
bootstrap
Cloud Init
#cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d
http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html
#!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t
m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt
chef/puppet
familiar tools
LSF
Grid Engine
Bright Cluster Manager
combining worlds
MIT Starcluster
$ starcluster start mycluster $ starcluster listclusters
http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
None
Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB
Public Dataset Provisioned Stack = Submit jobs to Grid Engine
None
None
None
Image: Chris Dagdigian