Field Notes from Expeditions in the Cloud

Field Notes from Expeditions in the Cloud

Keynote from Cloud Expo 2014, in NYC.

39488f9d172ab92fd352f2cd7b73258d?s=128

Matt Wood

June 10, 2014
Tweet

Transcript

  1. from Dr. Matt Wood matthew@amazon.com General Manager, Data Science a

    presentation by Matt Wood N E S W @mza FIELD NOTES EXPEDITIONS IN THE CLOUD
  2. RETAIL 280

  3. RETAIL MERCH 280

  4. RETAIL MERCH AWS 280

  5. 14th March 2006 S3 25th AUGUST 2006 EC2 22nd OCTOBER

    2009 RDS 15th FEBRUARY 2013 Redshift 18th JANUARY 2012 DYNAMODB RETAIL MERCH AWS 280
  6. 14th March 2006 S3 25th AUGUST 2006 EC2 22nd OCTOBER

    2009 RDS 15th FEBRUARY 2013 Redshift 18th JANUARY 2012 DYNAMODB RETAIL MERCH AWS 2008 2009 2010 2011 2012 2013 159 82 61 48 24 280
  7. 14th March 2006 S3 25th AUGUST 2006 EC2 22nd OCTOBER

    2009 RDS 15th FEBRUARY 2013 Redshift 170 launches in 2014 18th JANUARY 2012 DYNAMODB RETAIL MERCH AWS 2008 2009 2010 2011 2012 2013 159 82 61 48 24 280
  8. LOTS OF DATA

  9. LOTS OF DATA LOTS OF USES

  10. LOTS OF DATA LOTS OF USES LOTS OF USERS

  11. LOTS OF DATA LOTS OF USES LOTS OF USERS LOTS

    OF LOCATIONS
  12. LOTS OF DATA LOTS OF USES LOTS OF USERS LOTS

    OF LOCATIONS COMPLEXITY
  13. SOLAR PANELS Notes on CAN we make A better solar

    panel?” “
  14. SOLAR PANELS Notes on CAN we make A better solar

    panel?” “ Synthesize PURIFY ANALYZE PHOTO CELL?
  15. SOLAR PANELS Notes on CAN we make A better solar

    panel?” “ Synthesize PURIFY ANALYZE PHOTO CELL? GRAD STUDENT EQUIPMENT +
  16. SOLAR PANELS Notes on CAN we make A better solar

    panel?” “ 205,000 organic molecules Synthesize PURIFY ANALYZE PHOTO CELL? GRAD STUDENT EQUIPMENT +
  17. None
  18. 1.21 PFLOPS 16,788 instances

  19. 1.21 PFLOPS 16,788 instances 264 years of compute time <18

    hours
  20. 1.21 PFLOPS 16,788 instances 264 years of compute time <18

    hours $33k
  21. 1.21 PFLOPS 16,788 instances 264 years of compute time <18

    hours $33k COST MODELS ENABLE INNOVATION UTILIZATION
  22. 1.21 PFLOPS 16,788 instances 264 years of compute time <18

    hours $33k COST MODELS ENABLE INNOVATION UTILIZATION RESERVED CAPACITY
  23. 1.21 PFLOPS 16,788 instances 264 years of compute time <18

    hours $33k COST MODELS ENABLE INNOVATION UTILIZATION RESERVED CAPACITY ON DEMAND
  24. 1.21 PFLOPS 16,788 instances 264 years of compute time <18

    hours $33k COST MODELS ENABLE INNOVATION UTILIZATION RESERVED CAPACITY ON DEMAND
  25. VIDEO GAMES Notes on

  26. VIDEO GAMES Notes on “WHO BUYS VIDEO GAMES?”

  27. VIDEO GAMES Notes on “WHO BUYS VIDEO GAMES?” 3.5 billion

    records 13 TB of click stream logs 71 million unique cookies C luster
  28. VIDEO GAMES Notes on “WHO BUYS VIDEO GAMES?” 3.5 billion

    records 13 TB of click stream logs 71 million unique cookies C luster HUGE TVs
  29. 500% return on AD SPEND 17000% decrease in procurement time

  30. RETARGETING Notes on

  31. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE

    AD?” ”
  32. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE

    AD?” ” AD SERVERS
  33. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE

    AD?” ” AD SERVERS BIDDING ENGINE
  34. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE

    AD?” ” AD SERVERS BIDDING ENGINE DYNAMODB
  35. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE

    AD?” ” AD SERVERS BIDDING ENGINE 10ms 20MS 20MS 20MS HARD LIMIT: 100ms 10k advertisers 100 countries 1.5PB in S3 DYNAMODB
  36. 100 countries

  37. 100 countries

  38. 100 countries HUNDREDS OF MILLIONS OF RECORDS 10ks WRITES PER

    SECOND in sync at WITH APACHE STORM
  39. DYNAMODB THROUGHPUT AVERAGE LATENCY 11/1/2013 11/7/2013

  40. DYNAMODB THROUGHPUT AVERAGE LATENCY # REQUESTS 11/1/2013 11/7/2013

  41. DYNAMODB THROUGHPUT AVERAGE LATENCY # REQUESTS “THE LAW OF LARGE

    NUMBERS IS YOUR FRIEND UNTIL YOU REACH LARGE NUMBERS” 11/1/2013 11/7/2013
  42. DYNAMODB THROUGHPUT “THE LAW OF LARGE NUMBERS IS YOUR FRIEND

    UNTIL YOU REACH LARGE NUMBERS” PRECISE DESCRIPTION FORMAL METHODS TEST, TEST, TEST AVERAGE LATENCY # REQUESTS 11/1/2013 11/7/2013
  43. SOCIAL GAMINg Notes on

  44. SOCIAL GAMINg Notes on CAN PHYSICAL AND VIRTUAL WORLDS INTERACT?”

    ” SOCIAL EXPERIENCE BUILT RIGHT INTO THE GAMES
  45. SOCIAL GAMINg Notes on CAN PHYSICAL AND VIRTUAL WORLDS INTERACT?”

    ” SOCIAL EXPERIENCE BUILT RIGHT INTO THE GAMES LOW LATENCY HIGH AVAILABILITY ANALYTICS
  46. JAPAN US EUROPE

  47. JAPAN US EUROPE S3 CLOUDFRONT VPC

  48. JAPAN US EUROPE S3 CLOUDFRONT VPC EC2 EC2 EC2 AZ1

    AZ2 AZ3
  49. JAPAN US EUROPE REDSHIFT S3 CLOUDFRONT VPC EC2 EC2 EC2

    AZ1 AZ2 AZ3
  50. GENOMICS Notes on

  51. Chromosome 11 GENOMICS Notes on

  52. Chromosome 11 Chromosome X GENOMICS Notes on

  53. Chromosome 11 Chromosome X GENOMICS Notes on Chromosome 19

  54. Chromosome 11 Chromosome X Chromosome 15 GENOMICS Notes on Chromosome

    19
  55. Chromosome 11 Chromosome X Chromosome 15 GENOMICS Notes on Chromosome

    19
  56. Chromosome 11 Chromosome X Chromosome 15 GENOMICS Notes on Chromosome

    19
  57. DNA SEQUENCING COSTS < $1000 PER GENOME

  58. DNA SEQUENCING COSTS < $1000 PER GENOME

  59. DNA SEQUENCING COSTS < $1000 PER GENOME S3 COLLABORATION and

    TOOLING
  60. None
  61. CLINICAL TRIALs Notes on

  62. CLINICAL TRIALs Notes on BUSINESS DRIVERS Speeding server provisioning for

    R&D apps Extending capacity for internal grid environments Slowing internally hosted compute infrastructure growth On-boarding security, validation services and compliance Reducing cost while extending capabilities
  63. CLINICAL TRIAL SIMULATIONS Individual Clinical Trial Simulation Run Time (Min)

    Total number of trials Number of servers Number of CPUS TOTAL ANALYSIS RUN TIME (hours) COST INTERNAL AWS 56 56 2000 2000 2 32 60 ??? $336 1.2 2048 256
  64. REDUCED BURDEN ON pEDIATRIC SUBJECTS NUmber of subjects NUmber of

    blood samples per subject length of stay per subject length of study total study cost traditional DESIGN DESIGN OPTIMIZED USING CLINICAL TRIAL SIMULATION 60 40 12 5 72 hours 2.5 years $700k $250k 1.7 years 26 hours
  65. CONNECTED DEVICES GENOMICS CLINICAL TRIALS GAMING MOBILE DEVELOPMENT ADVERTISING HOSPITALITY

    RETAIL E-COMMERCE MANUFACTURING TRAVEL FREE STEAK SOCIAL APPS FINANCIAL ANALYTICS MEDIA STREAMING DISASTER RECOVERY ARCHIVAL INTERACTIVE TV APPS WEB APPS MARS EXPLORATION
  66. WHERE TO FOCUS? EARLY STAGE START-UP BILLION DOLLAR MULTINATIONAL

  67. AGILITY REASON #1:

  68. AGILITY EARLY STAGE START-UP REASON #1:

  69. AGILITY EARLY STAGE START-UP BILLION DOLLAR MULTINATIONAL REASON #1:

  70. FLY WHEELS FOR AGILITY SECURITY SCALE PERFORMANCE RELIABILITY COST

  71. ADOPTION PATTERNS OF How are enterprises using the cloud in

    2014?
  72. DEVELOPMENT & TESTING SHAREPOINT & SAP ORACLE SAP

  73. NEW WORKLOADS GLOBAL DEALS ENGINE PRODUCT PROTOTYPING MOBILE MUSIC DISCOVERY

    HOTEL BOOKING VIDEO STREAMING
  74. SUPPLEMENT EXISTING WORKFLOWS operational applications equipment leasing analytics with the

    cloud
  75. SUPPLEMENT EXISTING WORKFLOWS with on-premises infrastructure AWS serves up application

    content & data Integration back to Samsung Data Centers for financial transactions
  76. MIGRATING EXISTING APPLICATIONS 500 web properties in 5 months CLINICAL

    TRIALS 1,700 web properties
  77. ALL IN “WE HAVE TO BE GREAT AT A NUMBER

    OF THINGS... OPERATING DATA CENTERS IS NOT ONE OF THOSE THINGS” “WE ARE A HOSPITALITY MANAGEMENT COMPANY, NOT AN IT SERVICES COMPANY” “FRIENDS DON’T LET FRIENDS BUILD DATA CENTERS” INFOR CLOUDSUITE micro-vertical BUSINESS APPS
  78. IN SUMMARY... DATA DRIVES INNOVATION WHEN COMBINED WITH UTILITY COMPUTING

    Broad and deep services enable new business AGILITY REMAINS THE #1 REASON FOR ADOPTING THE AWS CLOUD ENTERPRISES LEAD WITh DEV, TEST & NEW WORKLOADS MIGRATION AND SUPPLEMENTS QUICKLY FOLLOW Increasingly ‘ALL IN’
  79. @mza MATTHEW@AMAZON.COM THANK YOU

  80. More sessions from AWS today 11.00am: Transforming Applications into SaaS

    11.40am: Big Data Is Best Served Fresh: Real-time Analytics in the Cloud 1.55pm: What is the Internet of Things – and how can it help my business? 4.00pm: Moving Enterprises to the Cloud the Right Way and for the Right Reasons 4.40pm: Creating a Cloud Based DR Solution for your Enterprise