Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Intel Theater Presentation @ SC11
Search
Deepak Singh
November 19, 2011
Technology
6
180
Intel Theater Presentation @ SC11
Presented at the Intel Theater at SC11
Deepak Singh
November 19, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
100
Platforms for scientific data analysis
mndoci
3
100
FGED Keynote
mndoci
3
92
Open Mic Science - May 7, 2012
mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
250
A Platform for Data Science
mndoci
6
14k
Talk at West Coast Association of Shared Directors meeting
mndoci
3
150
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
110
Platforms for Data Science
mndoci
3
200
Other Decks in Technology
See All in Technology
60以上のプロダクトを持つ組織における開発者体験向上への取り組み - チームAPIとBackstageで構築する組織の可視化基盤 - / sre next 2025 Efforts to Improve Developer Experience in an Organization with Over 60 Products
vtryo
0
150
Lazy application authentication with Tailscale
bluehatbrit
0
220
高速なプロダクト開発を実現、創業期から掲げるエンタープライズアーキテクチャ
kawauso
3
9.6k
MobileActOsaka_250704.pdf
akaitadaaki
0
150
Delegating the chores of authenticating users to Keycloak
ahus1
0
160
CRE Camp #1 エンジニアリングを民主化するCREチームでありたい話
mntsq
1
130
ビズリーチが挑む メトリクスを活用した技術的負債の解消 / dev-productivity-con2025
visional_engineering_and_design
3
7.8k
[ JAWS-UG千葉支部 x 彩の国埼玉支部 ]ムダ遣い卒業!FinOpsで始めるAWSコスト最適化の第一歩
sh_fk2
2
100
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
960
無意味な開発生産性の議論から抜け出すための予兆検知とお金とAI
i35_267
6
13k
Enhancing SaaS Product Reliability and Release Velocity through Optimized Testing Approach
ropqa
1
240
「クラウドコスト絶対削減」を支える技術—FinOpsを超えた徹底的なクラウドコスト削減の実践論
delta_tech
4
170
Featured
See All Featured
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Building an army of robots
kneath
306
45k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Raft: Consensus for Rubyists
vanstee
140
7k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Fireside Chat
paigeccino
37
3.5k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
Being A Developer After 40
akosma
90
590k
How to Ace a Technical Interview
jacobian
278
23k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Transcript
HPC with Amazon EC2 Deepak Singh @mndoci P r i
n c i p a l P r o d u c t M a n a g e r
Amazon Web Services
4
2
1. Infrastructure
None
ec2-run-instances
None
secure global on demand
programmable
None
None
None
elastic
None
instance types
standard (m1) high memory (m2) high CPU (c1) t1.micro
high performance
“Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1
billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
cluster computing
MPI
bandwidth intensive
Cluster Compute Instance
2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7
TB disk HVM cc1.4xlarge
10 gig E
Placement Group
Placement group full- bisection
linpack
Cores 7040 Rmax 41.82 Rpeak 82.51
231 November 2010
451 June 2011
WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)
Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
New Cluster Compute Instance
2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB
disk HVM cc2.8xlarge
linpack
Cores 17024 Rmax 240.09 Rpeak 354.12
42 November 2011
optimizing costs
on-demand
reserved
spot
None
None
None
30,472 cores
$1279/hr
2. Orchestration
None
AWS CloudFormation
bootstrap
Cloud Init
#cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d
http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html
#!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t
m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt
chef/puppet
familiar tools
LSF
Grid Engine
Bright Cluster Manager
combining worlds
MIT Starcluster
$ starcluster start mycluster $ starcluster listclusters
http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
None
Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB
Public Dataset Provisioned Stack = Submit jobs to Grid Engine
None
None
None
Image: Chris Dagdigian