Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intel Theater Presentation @ SC11

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Deepak Singh Deepak Singh
November 19, 2011
Technology
210
6

Intel Theater Presentation @ SC11

Presented at the Intel Theater at SC11

Avatar for Deepak Singh

Deepak Singh

November 19, 2011

More Decks by Deepak Singh

See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
Avatar for Deepak Singh mndoci
2
130
Platforms for scientific data analysis
Avatar for Deepak Singh mndoci
3
120
FGED Keynote
Avatar for Deepak Singh mndoci
3
110
Open Mic Science - May 7, 2012
Avatar for Deepak Singh mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
Avatar for Deepak Singh mndoci
1
270
A Platform for Data Science
Avatar for Deepak Singh mndoci
6
15k
Talk at West Coast Association of Shared Directors meeting
Avatar for Deepak Singh mndoci
3
160
A platform for data science - Systems Bioinformatics Workshop
Avatar for Deepak Singh mndoci
3
130
Platforms for Data Science
Avatar for Deepak Singh mndoci
3
210

Other Decks in Technology

See All in Technology
2026-08-15 JAWS-UG 茨城 #16 Secrets ManagerにおけるSecret値の管理(Terraformの場合) / Secrets Manager Secrets
Avatar for SUZUKI Masashi masasuzu
0
390
Kiro入門｜仕様駆動開発で変わるAI時代の開発スタイル
Avatar for 工藤皓夢 cmkudo
0
250
LLM・AIエージェントシステムベストプラクティス
Avatar for shibuiwilliam shibuiwilliam
6
1.6k
도구에서 동료까지: 10년차 AI 스타트업의 AI 적응기
Avatar for Jeongkyu Shin inureyes
PRO
1
290
Go 1.27 の標準パッケージに uuid が入った！のでいろいろ喋る / go_127_std_go_uuid
Avatar for convto convto
3
540
Head First モブプログラミング / Head First Mobprogramming
Avatar for TAKAKING22 takaking22
10
12k
AIに持続⼒を与える 判断の⻑期記憶設計
Avatar for eiei114 eiei114
1
420
国家プロジェクトを支える「さくらONE」 大規模LLM開発におけるGPU障害を乗り越えるクラスター運用戦略
Avatar for GPU UNITE gpuunite_official
0
150
Bits Agent Builder の⼊⾨と活⽤事例
Avatar for 株式会社ヌーラボ nulabinc
PRO
0
250
ホームラボ紹介
Avatar for 世良泰明 y_sera15
0
220
AI・HPC開発を支えるGPU環境の新しい選択肢 液冷GPUシステム「AquSys」の取り組み
Avatar for GPU UNITE gpuunite_official
0
150
まちスペース®とデジタルツインと「まちづくり」
Avatar for hiro_OGI hiro_ogi
0
130

Featured

See All Featured
Everyday Curiosity
Avatar for Cassini Nazir cassininazir
0
290
Paper Plane (Part 1)
Avatar for Katie Cordina Lindsey katiecoart
PRO
1
10k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
82
6.5k
svc-hook: hooking system calls on ARM64 by binary rewriting
Avatar for Akira Moroo retrage
2
500
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
4.4k
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
57
7.1k
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
2
480
ラッコキーワード サービス紹介資料
Avatar for ラッコ株式会社 rakko
1
4.4M
16th Malabo Montpellier Forum Presentation
Avatar for AKADEMIYA2063 akademiya2063
PRO
0
350
A better future with KSS
Avatar for Kyle Neath kneath
240
18k
Game over? The fight for quality and originality in the time of robots
Avatar for Wayne Barker wayneb77
1
250
Navigating Team Friction
Avatar for Lara Hogan lara
192
16k

Transcript

  1. HPC with Amazon EC2 Deepak Singh @mndoci P r i

    n c i p a l P r o d u c t M a n a g e r

  2. Amazon Web Services

  3. 4

  4. 2

  5. 1. Infrastructure

  6. None

  7. ec2-run-instances

  8. None

  9. secure global on demand

  10. programmable

  11. None
  12. None
  13. None

  14. elastic

  15. None

  16. instance types

  17. standard (m1) high memory (m2) high CPU (c1) t1.micro

  18. high performance

  19. “Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1

    billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets

  20. cluster computing

  21. MPI

  22. bandwidth intensive

  23. Cluster Compute Instance

  24. 2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7

    TB disk HVM cc1.4xlarge

  25. 10 gig E

  26. Placement Group

  27. Placement group full- bisection

  28. linpack

  29. Cores 7040 Rmax 41.82 Rpeak 82.51

  30. 231 November 2010

  31. 451 June 2011

  32. WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)

    Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)

  33. New Cluster Compute Instance

  34. 2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB

    disk HVM cc2.8xlarge

  35. linpack

  36. Cores 17024 Rmax 240.09 Rpeak 354.12

  37. 42 November 2011

  38. optimizing costs

  39. on-demand

  40. reserved

  41. spot

  42. None
  43. None
  44. None

  45. 30,472 cores

  46. $1279/hr

  47. 2. Orchestration

  48. None

  49. AWS CloudFormation

  50. bootstrap

  51. Cloud Init

  52. #cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d

    http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html

  53. #!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t

    m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt

  54. chef/puppet

  55. familiar tools

  56. LSF

  57. Grid Engine

  58. Bright Cluster Manager

  59. combining worlds

  60. MIT Starcluster

  61. $ starcluster start mycluster $ starcluster listclusters

  62. http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/

  63. None

  64. Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB

    Public Dataset Provisioned Stack = Submit jobs to Grid Engine
  65. None
  66. None
  67. None

  68. Image: Chris Dagdigian