Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Intel Theater Presentation @ SC11
Search
Deepak Singh
November 19, 2011
Technology
6
170
Intel Theater Presentation @ SC11
Presented at the Intel Theater at SC11
Deepak Singh
November 19, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
97
Platforms for scientific data analysis
mndoci
3
87
FGED Keynote
mndoci
3
88
Open Mic Science - May 7, 2012
mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
250
A Platform for Data Science
mndoci
6
14k
Talk at West Coast Association of Shared Directors meeting
mndoci
3
150
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
100
Platforms for Data Science
mndoci
3
190
Other Decks in Technology
See All in Technology
Global Databaseで実現するマルチリージョン自動切替とBlue/Greenデプロイ
j2yano
0
160
スクラムというコンフォートゾーンから抜け出そう!プロジェクト全体に目を向けるインセプションデッキ / Inception Deck for seeing the whole project
takaking22
3
130
Exadata Database Service on Cloud@Customer セキュリティ、ネットワーク、および管理について
oracle4engineer
PRO
2
1.6k
目標と時間軸 〜ベイビーステップでケイパビリティを高めよう〜
kakehashi
PRO
8
1k
手を動かしてレベルアップしよう!
maruto
0
250
OPENLOGI Company Profile
hr01
0
60k
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
540
JavaにおけるNull非許容性
skrb
2
2.7k
Qiita Organizationを導入したら、アウトプッターが爆増して会社がちょっと有名になった件
minorun365
PRO
1
320
OCI Success Journey OCIの何が評価されてる?疑問に答える事例セミナー(2025年2月実施)
oracle4engineer
PRO
2
220
データモデルYANGの処理系を再発明した話
tjmtrhs
0
310
Platform Engineeringで クラウドの「楽しくない」を解消しよう
jacopen
4
210
Featured
See All Featured
Building Adaptive Systems
keathley
40
2.4k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
175
52k
Large-scale JavaScript Application Architecture
addyosmani
511
110k
Practical Orchestrator
shlominoach
186
10k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.3k
KATA
mclloyd
29
14k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.2k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Mobile First: as difficult as doing things right
swwweet
223
9.5k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
Building Your Own Lightsaber
phodgson
104
6.2k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
100
18k
Transcript
HPC with Amazon EC2 Deepak Singh @mndoci P r i
n c i p a l P r o d u c t M a n a g e r
Amazon Web Services
4
2
1. Infrastructure
None
ec2-run-instances
None
secure global on demand
programmable
None
None
None
elastic
None
instance types
standard (m1) high memory (m2) high CPU (c1) t1.micro
high performance
“Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1
billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
cluster computing
MPI
bandwidth intensive
Cluster Compute Instance
2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7
TB disk HVM cc1.4xlarge
10 gig E
Placement Group
Placement group full- bisection
linpack
Cores 7040 Rmax 41.82 Rpeak 82.51
231 November 2010
451 June 2011
WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)
Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
New Cluster Compute Instance
2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB
disk HVM cc2.8xlarge
linpack
Cores 17024 Rmax 240.09 Rpeak 354.12
42 November 2011
optimizing costs
on-demand
reserved
spot
None
None
None
30,472 cores
$1279/hr
2. Orchestration
None
AWS CloudFormation
bootstrap
Cloud Init
#cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d
http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html
#!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t
m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt
chef/puppet
familiar tools
LSF
Grid Engine
Bright Cluster Manager
combining worlds
MIT Starcluster
$ starcluster start mycluster $ starcluster listclusters
http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
None
Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB
Public Dataset Provisioned Stack = Submit jobs to Grid Engine
None
None
None
Image: Chris Dagdigian