Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Intel Theater Presentation @ SC11
Search
Deepak Singh
November 19, 2011
Technology
6
160
Intel Theater Presentation @ SC11
Presented at the Intel Theater at SC11
Deepak Singh
November 19, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
92
Platforms for scientific data analysis
mndoci
3
79
FGED Keynote
mndoci
3
68
Open Mic Science - May 7, 2012
mndoci
4
1.2k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
240
A Platform for Data Science
mndoci
6
13k
Talk at West Coast Association of Shared Directors meeting
mndoci
3
140
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
99
Platforms for Data Science
mndoci
3
180
Other Decks in Technology
See All in Technology
Building Dashboards as a Hobby
egmc
0
140
「スニダン」開発組織の構造に込めた意図 ~組織作りはパッションや政治ではない!~
rinchsan
3
560
TechFeed Experts Night#27 〜 フロントエンドフレームワーク最前線 (Svelte)
baseballyama
1
470
開発パフォーマンスを最大化するための開発体制
ham0215
2
380
ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation
lmi
1
120
Yahoo! 知恵袋フロントエンドをリアーキテクトしている話
lycorptech_jp
PRO
1
130
ServiceNow Knowledge Learning Rise up
manarobot
0
210
Compose Compiler Metricsを使った実践的なコードレビュー
tomorrowkey
1
220
Meta Quest 3 で動く桜マシマシ WebXR アプリを IBM Cloud Code Engine と Babylon.js で作った話
1ftseabass
PRO
0
120
エンジニア候補者向け資料2024.04.24.pdf
macloud
0
3.3k
自己改善からチームを動かす! 「セルフエンジニアリングマネージャー」のすゝめ
shoota
6
480
GraphQL 成熟度モデルの紹介と、プロダクトに当てはめた事例 / GraphQL maturity model
mh4gf
7
1.3k
Featured
See All Featured
How STYLIGHT went responsive
nonsquared
92
4.8k
The Language of Interfaces
destraynor
151
23k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
2
3.4k
Visualization
eitanlees
136
14k
Faster Mobile Websites
deanohume
299
30k
The Art of Programming - Codeland 2020
erikaheidi
42
12k
Creatively Recalculating Your Daily Design Routine
revolveconf
210
11k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
25
2.3k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
2
1.3k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
40
4.4k
Adopting Sorbet at Scale
ufuk
68
8.6k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
Transcript
HPC with Amazon EC2 Deepak Singh @mndoci P r i
n c i p a l P r o d u c t M a n a g e r
Amazon Web Services
4
2
1. Infrastructure
None
ec2-run-instances
None
secure global on demand
programmable
None
None
None
elastic
None
instance types
standard (m1) high memory (m2) high CPU (c1) t1.micro
high performance
“Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1
billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
cluster computing
MPI
bandwidth intensive
Cluster Compute Instance
2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7
TB disk HVM cc1.4xlarge
10 gig E
Placement Group
Placement group full- bisection
linpack
Cores 7040 Rmax 41.82 Rpeak 82.51
231 November 2010
451 June 2011
WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)
Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
New Cluster Compute Instance
2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB
disk HVM cc2.8xlarge
linpack
Cores 17024 Rmax 240.09 Rpeak 354.12
42 November 2011
optimizing costs
on-demand
reserved
spot
None
None
None
30,472 cores
$1279/hr
2. Orchestration
None
AWS CloudFormation
bootstrap
Cloud Init
#cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d
http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html
#!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t
m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt
chef/puppet
familiar tools
LSF
Grid Engine
Bright Cluster Manager
combining worlds
MIT Starcluster
$ starcluster start mycluster $ starcluster listclusters
http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
None
Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB
Public Dataset Provisioned Stack = Submit jobs to Grid Engine
None
None
None
Image: Chris Dagdigian