Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Intel Theater Presentation @ SC11
Search
Deepak Singh
November 19, 2011
Technology
6
160
Intel Theater Presentation @ SC11
Presented at the Intel Theater at SC11
Deepak Singh
November 19, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
92
Platforms for scientific data analysis
mndoci
3
79
FGED Keynote
mndoci
3
68
Open Mic Science - May 7, 2012
mndoci
4
1.2k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
240
A Platform for Data Science
mndoci
6
13k
Talk at West Coast Association of Shared Directors meeting
mndoci
3
140
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
99
Platforms for Data Science
mndoci
3
180
Other Decks in Technology
See All in Technology
ExaDB-D dbaascli で出来ること
oracle4engineer
PRO
0
2.1k
.NET Profiler in 2024.
kkamegawa
2
1.7k
Cracking the KubeCon CfP
inductor
2
270
生産性向上チームの紹介
cybozuinsideout
PRO
1
930
Building a RAG-poweredAI chat appwith Python and VS Code
pamelafox
0
160
Azureの基本的な権限管理の勉強会
yhana
1
2.1k
Amplify 🩷 Bedrock 〜生成AI入門〜
minorun365
PRO
8
900
20分で完全に理解するGrafanaダッシュボード
hamadakoji
5
910
【SORACOM UG 東海】あらゆるモノがつながる社会へ、IoT と SORACOM
soracom
PRO
1
150
Autonomous Database Cloud 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
15
35k
Gitlab本から学んだこと - そーだいなるプレイバック / gitlab-book
soudai
7
1.3k
Cypress or Playwright?
rainerhahnekamp
0
170
Featured
See All Featured
Teambox: Starting and Learning
jrom
128
8.4k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
20
1.8k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.1k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
20
1.6k
Mobile First: as difficult as doing things right
swwweet
217
8.6k
Automating Front-end Workflow
addyosmani
1357
200k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
275
13k
Build The Right Thing And Hit Your Dates
maggiecrowley
25
2k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
6.9k
Git: the NoSQL Database
bkeepers
PRO
423
63k
Being A Developer After 40
akosma
66
580k
Gamification - CAS2011
davidbonilla
77
4.6k
Transcript
HPC with Amazon EC2 Deepak Singh @mndoci P r i
n c i p a l P r o d u c t M a n a g e r
Amazon Web Services
4
2
1. Infrastructure
None
ec2-run-instances
None
secure global on demand
programmable
None
None
None
elastic
None
instance types
standard (m1) high memory (m2) high CPU (c1) t1.micro
high performance
“Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1
billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
cluster computing
MPI
bandwidth intensive
Cluster Compute Instance
2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7
TB disk HVM cc1.4xlarge
10 gig E
Placement Group
Placement group full- bisection
linpack
Cores 7040 Rmax 41.82 Rpeak 82.51
231 November 2010
451 June 2011
WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)
Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
New Cluster Compute Instance
2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB
disk HVM cc2.8xlarge
linpack
Cores 17024 Rmax 240.09 Rpeak 354.12
42 November 2011
optimizing costs
on-demand
reserved
spot
None
None
None
30,472 cores
$1279/hr
2. Orchestration
None
AWS CloudFormation
bootstrap
Cloud Init
#cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d
http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html
#!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t
m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt
chef/puppet
familiar tools
LSF
Grid Engine
Bright Cluster Manager
combining worlds
MIT Starcluster
$ starcluster start mycluster $ starcluster listclusters
http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
None
Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB
Public Dataset Provisioned Stack = Submit jobs to Grid Engine
None
None
None
Image: Chris Dagdigian