Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intel Theater Presentation @ SC11

Avatar for Deepak Singh Deepak Singh
November 19, 2011
Technology
210
6

Intel Theater Presentation @ SC11

Presented at the Intel Theater at SC11

Avatar for Deepak Singh

Deepak Singh

November 19, 2011

More Decks by Deepak Singh

See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
Avatar for Deepak Singh mndoci
2
130
Platforms for scientific data analysis
Avatar for Deepak Singh mndoci
3
120
FGED Keynote
Avatar for Deepak Singh mndoci
3
110
Open Mic Science - May 7, 2012
Avatar for Deepak Singh mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
Avatar for Deepak Singh mndoci
1
270
A Platform for Data Science
Avatar for Deepak Singh mndoci
6
15k
Talk at West Coast Association of Shared Directors meeting
Avatar for Deepak Singh mndoci
3
160
A platform for data science - Systems Bioinformatics Workshop
Avatar for Deepak Singh mndoci
3
130
Platforms for Data Science
Avatar for Deepak Singh mndoci
3
210

Other Decks in Technology

See All in Technology
Hatena Engineer Seminar 37 jj1uzh
Avatar for jj1uzh jj1uzh
0
150
現場のトークンマネジメント
Avatar for dak2 dak2
1
200
脱SaaS！FDEを支えるプロビジョニングと分離設計
Avatar for Kenichi SUZUKI knih
0
300
Flow 不死：AI 時代 DevOps 的不變本質
Avatar for Cheng-Wei Chen cheng_wei_chen
2
550
スタートアップにAmazon EKSは早すぎる？ マルチプロダクト戦略を加速する Platform Engineeringの実践 / Is Amazon EKS Too Soon for Startups? Practical Platform Engineering to Accelerate a Multi-Product Strategy
Avatar for elmo elmodev09
1
1.9k
#エンジニアBooks 30分でわかる
「技術記事を書く技術」 / engineer-books 2026-06-30
Avatar for Junichi Ito jnchito
1
130
UIパーツの設計を「型」から読み解く 〜TSKaigiのセッションから得た学び〜
Avatar for yud0uhu yud0uhu
0
100
そこにあるから地図ができる～位置を示す"モノ"を愉しむ～ - Interface 2026年6月号GPS特集オフ会 / interface_202606_GPS_offline
Avatar for sakaik sakaik
1
110
週末にループ・エンジニアリングの理解を深めるためのスライド
Avatar for 長津孝輔 nagatsu
0
570
BPaaSで進むAIオペレーションの現在地 AI実装が効く領域とスケーラビリティの選定と実装
Avatar for KentaroFujii kentarofujii
0
210
WebGIS AI Agentの紹介
Avatar for _shimizu _shimizu
0
580
AIペネトレーションテスト・ セキュリティ検証「AgenticSec」紹介資料
Avatar for Sho Nakatani laysakura
2
7.7k

Featured

See All Featured
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
85
9.5k
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
Avatar for Akash Hashmi akashhashmi
0
370
How To Speak Unicorn (iThemes Webinar)
Avatar for Michelle Schulp Hunt marktimemedia
1
490
Avoiding the “Bad Training, Faster” Trap in the Age of AI
Avatar for Mike Taylor tmiket
0
180
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
356
21k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.5k
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
Avatar for Aleyda Solis aleyda
1
2.1k
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
350
The browser strikes back
Avatar for Jono Alderson jonoalderson
0
1.3k
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
2
400
Odyssey Design
Avatar for Ryan Kendrick rkendrick25
PRO
2
710

Transcript

  1. HPC with Amazon EC2 Deepak Singh @mndoci P r i

    n c i p a l P r o d u c t M a n a g e r

  2. Amazon Web Services

  3. 4

  4. 2

  5. 1. Infrastructure

  6. None

  7. ec2-run-instances

  8. None

  9. secure global on demand

  10. programmable

  11. None
  12. None
  13. None

  14. elastic

  15. None

  16. instance types

  17. standard (m1) high memory (m2) high CPU (c1) t1.micro

  18. high performance

  19. “Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1

    billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets

  20. cluster computing

  21. MPI

  22. bandwidth intensive

  23. Cluster Compute Instance

  24. 2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7

    TB disk HVM cc1.4xlarge

  25. 10 gig E

  26. Placement Group

  27. Placement group full- bisection

  28. linpack

  29. Cores 7040 Rmax 41.82 Rpeak 82.51

  30. 231 November 2010

  31. 451 June 2011

  32. WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)

    Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)

  33. New Cluster Compute Instance

  34. 2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB

    disk HVM cc2.8xlarge

  35. linpack

  36. Cores 17024 Rmax 240.09 Rpeak 354.12

  37. 42 November 2011

  38. optimizing costs

  39. on-demand

  40. reserved

  41. spot

  42. None
  43. None
  44. None

  45. 30,472 cores

  46. $1279/hr

  47. 2. Orchestration

  48. None

  49. AWS CloudFormation

  50. bootstrap

  51. Cloud Init

  52. #cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d

    http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html

  53. #!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t

    m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt

  54. chef/puppet

  55. familiar tools

  56. LSF

  57. Grid Engine

  58. Bright Cluster Manager

  59. combining worlds

  60. MIT Starcluster

  61. $ starcluster start mycluster $ starcluster listclusters

  62. http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/

  63. None

  64. Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB

    Public Dataset Provisioned Stack = Submit jobs to Grid Engine
  65. None
  66. None
  67. None

  68. Image: Chris Dagdigian