Field Notes from Expeditions in the Cloud

Keynote from Cloud Expo 2014, in NYC.

Transcript

1. from
   Dr. Matt Wood
   matthew@amazon.com
   General Manager, Data Science
   a presentation by
   Matt Wood
   N
   E
   S
   W
   @mza
   FIELD NOTES
   EXPEDITIONS
   IN THE CLOUD
   

   View Slide

2. RETAIL
   280
   

   View Slide

3. RETAIL MERCH
   280
   

   View Slide

4. RETAIL MERCH AWS
   280
   

   View Slide

5. 14th March 2006 S3
   25th AUGUST 2006 EC2
   22nd OCTOBER 2009 RDS
   15th FEBRUARY 2013 Redshift
   18th JANUARY 2012 DYNAMODB
   RETAIL MERCH AWS
   280
   

   View Slide

6. 14th March 2006 S3
   25th AUGUST 2006 EC2
   22nd OCTOBER 2009 RDS
   15th FEBRUARY 2013 Redshift
   18th JANUARY 2012 DYNAMODB
   RETAIL MERCH AWS 2008
   2009
   2010
   2011
   2012
   2013
   159
   82
   61
   48
   24
   280
   

   View Slide

7. 14th March 2006 S3
   25th AUGUST 2006 EC2
   22nd OCTOBER 2009 RDS
   15th FEBRUARY 2013 Redshift 170 launches in 2014
   18th JANUARY 2012 DYNAMODB
   RETAIL MERCH AWS 2008
   2009
   2010
   2011
   2012
   2013
   159
   82
   61
   48
   24
   280
   

   View Slide

8. LOTS OF DATA
   

   View Slide

9. LOTS OF DATA
   LOTS OF USES
   

   View Slide

10. LOTS OF DATA
    LOTS OF USES
    LOTS OF USERS
    

    View Slide

11. LOTS OF DATA
    LOTS OF USES
    LOTS OF USERS
    LOTS OF LOCATIONS
    

    View Slide

12. LOTS OF DATA
    LOTS OF USES
    LOTS OF USERS
    LOTS OF LOCATIONS
    COMPLEXITY
    

    View Slide

13. SOLAR PANELS
    Notes on
    CAN we make A
    better solar panel?”
    “
    

    View Slide

14. SOLAR PANELS
    Notes on
    CAN we make A
    better solar panel?”
    “
    Synthesize
    PURIFY
    ANALYZE
    PHOTO CELL?
    

    View Slide

15. SOLAR PANELS
    Notes on
    CAN we make A
    better solar panel?”
    “
    Synthesize
    PURIFY
    ANALYZE
    PHOTO CELL?
    GRAD
    STUDENT
    EQUIPMENT
    +
    

    View Slide

16. SOLAR PANELS
    Notes on
    CAN we make A
    better solar panel?”
    “
    205,000
    organic molecules
    Synthesize
    PURIFY
    ANALYZE
    PHOTO CELL?
    GRAD
    STUDENT
    EQUIPMENT
    +
    

    View Slide

17. View Slide

18. 1.21 PFLOPS
    16,788 instances
    

    View Slide

19. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    

    View Slide

20. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k
    

    View Slide

21. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k
    COST MODELS
    ENABLE INNOVATION
    UTILIZATION
    

    View Slide

22. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k
    COST MODELS
    ENABLE INNOVATION
    UTILIZATION
    RESERVED CAPACITY
    

    View Slide

23. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k
    COST MODELS
    ENABLE INNOVATION
    UTILIZATION
    RESERVED CAPACITY
    ON DEMAND
    

    View Slide

24. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k
    COST MODELS
    ENABLE INNOVATION
    UTILIZATION
    RESERVED CAPACITY
    ON DEMAND
    

    View Slide

25. VIDEO GAMES
    Notes on
    

    View Slide

26. VIDEO GAMES
    Notes on
    “WHO BUYS VIDEO GAMES?”
    

    View Slide

27. VIDEO GAMES
    Notes on
    “WHO BUYS VIDEO GAMES?”
    3.5 billion records
    13 TB of click stream logs
    71 million unique cookies
    C
    luster
    

    View Slide

28. VIDEO GAMES
    Notes on
    “WHO BUYS VIDEO GAMES?”
    3.5 billion records
    13 TB of click stream logs
    71 million unique cookies
    C
    luster
    HUGE TVs
    

    View Slide

29. 500% return on
    AD SPEND
    17000% decrease in
    procurement time
    

    View Slide

30. RETARGETING
    Notes on
    

    View Slide

31. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”
    ”
    

    View Slide

32. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”
    ”
    AD SERVERS
    

    View Slide

33. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”
    ”
    AD SERVERS
    BIDDING ENGINE
    

    View Slide

34. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”
    ”
    AD SERVERS
    BIDDING ENGINE
    DYNAMODB
    

    View Slide

35. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”
    ”
    AD SERVERS
    BIDDING ENGINE
    10ms
    20MS
    20MS 20MS
    HARD LIMIT: 100ms
    10k advertisers
    100 countries
    1.5PB in S3
    DYNAMODB
    

    View Slide

36. 100 countries
    

    View Slide

37. 100 countries
    

    View Slide

38. 100 countries
    HUNDREDS OF MILLIONS
    OF RECORDS
    10ks WRITES PER SECOND
    in sync at
    WITH APACHE STORM
    

    View Slide

39. DYNAMODB THROUGHPUT
    AVERAGE LATENCY
    11/1/2013 11/7/2013
    

    View Slide

40. DYNAMODB THROUGHPUT
    AVERAGE LATENCY
    # REQUESTS
    11/1/2013 11/7/2013
    

    View Slide

41. DYNAMODB THROUGHPUT
    AVERAGE LATENCY
    # REQUESTS
    “THE LAW OF LARGE
    NUMBERS IS YOUR FRIEND
    UNTIL YOU REACH
    LARGE NUMBERS”
    11/1/2013 11/7/2013
    

    View Slide

42. DYNAMODB THROUGHPUT
    “THE LAW OF LARGE
    NUMBERS IS YOUR FRIEND
    UNTIL YOU REACH
    LARGE NUMBERS”
    PRECISE DESCRIPTION
    FORMAL METHODS
    TEST, TEST, TEST
    AVERAGE LATENCY
    # REQUESTS
    11/1/2013 11/7/2013
    

    View Slide

43. SOCIAL GAMINg
    Notes on
    

    View Slide

44. SOCIAL GAMINg
    Notes on
    CAN PHYSICAL AND VIRTUAL
    WORLDS INTERACT?”
    ”
    SOCIAL EXPERIENCE
    BUILT RIGHT INTO THE GAMES
    

    View Slide

45. SOCIAL GAMINg
    Notes on
    CAN PHYSICAL AND VIRTUAL
    WORLDS INTERACT?”
    ”
    SOCIAL EXPERIENCE
    BUILT RIGHT INTO THE GAMES
    LOW LATENCY
    HIGH AVAILABILITY
    ANALYTICS
    

    View Slide

46. JAPAN
    US EUROPE
    

    View Slide

47. JAPAN
    US EUROPE
    S3 CLOUDFRONT
    VPC
    

    View Slide

48. JAPAN
    US EUROPE
    S3 CLOUDFRONT
    VPC
    EC2 EC2 EC2
    AZ1 AZ2 AZ3
    

    View Slide

49. JAPAN
    US EUROPE
    REDSHIFT
    S3 CLOUDFRONT
    VPC
    EC2 EC2 EC2
    AZ1 AZ2 AZ3
    

    View Slide

50. GENOMICS
    Notes on
    

    View Slide

51. Chromosome 11 GENOMICS
    Notes on
    

    View Slide

52. Chromosome 11
    Chromosome X
    GENOMICS
    Notes on
    

    View Slide

53. Chromosome 11
    Chromosome X
    GENOMICS
    Notes on
    Chromosome 19
    

    View Slide

54. Chromosome 11
    Chromosome X
    Chromosome 15
    GENOMICS
    Notes on
    Chromosome 19
    

    View Slide

55. Chromosome 11
    Chromosome X
    Chromosome 15
    GENOMICS
    Notes on
    Chromosome 19
    

    View Slide

56. Chromosome 11
    Chromosome X
    Chromosome 15
    GENOMICS
    Notes on
    Chromosome 19
    

    View Slide

57. DNA SEQUENCING COSTS
    < $1000
    PER GENOME
    

    View Slide

58. DNA SEQUENCING COSTS
    < $1000
    PER GENOME
    

    View Slide

59. DNA SEQUENCING COSTS
    < $1000
    PER GENOME
    S3
    COLLABORATION and TOOLING
    

    View Slide

60. View Slide

61. CLINICAL TRIALs
    Notes on
    

    View Slide

62. CLINICAL TRIALs
    Notes on
    BUSINESS DRIVERS
    Speeding server provisioning for R&D apps
    Extending capacity for internal grid
    environments
    Slowing internally hosted compute
    infrastructure growth
    On-boarding security, validation services and
    compliance
    Reducing cost while extending
    capabilities
    

    View Slide

63. CLINICAL TRIAL SIMULATIONS
    Individual Clinical Trial Simulation Run Time (Min)
    Total number of trials
    Number of servers
    Number of CPUS
    TOTAL ANALYSIS RUN TIME (hours)
    COST
    INTERNAL AWS
    56 56
    2000 2000
    2
    32
    60
    ??? $336
    1.2
    2048
    256
    

    View Slide

64. REDUCED BURDEN ON
    pEDIATRIC SUBJECTS
    NUmber of subjects
    NUmber of blood samples per subject
    length of stay per subject
    length of study
    total study cost
    traditional
    DESIGN
    DESIGN OPTIMIZED
    USING CLINICAL
    TRIAL SIMULATION
    60 40
    12 5
    72 hours
    2.5 years
    $700k $250k
    1.7 years
    26 hours
    

    View Slide

65. CONNECTED
    DEVICES
    GENOMICS
    CLINICAL
    TRIALS
    GAMING
    MOBILE
    DEVELOPMENT
    ADVERTISING
    HOSPITALITY
    RETAIL
    E-COMMERCE MANUFACTURING
    TRAVEL
    FREE STEAK
    SOCIAL APPS
    FINANCIAL
    ANALYTICS
    MEDIA STREAMING
    DISASTER
    RECOVERY
    ARCHIVAL
    INTERACTIVE
    TV APPS
    WEB APPS
    MARS
    EXPLORATION
    

    View Slide

66. WHERE TO
    FOCUS?
    EARLY STAGE
    START-UP
    BILLION DOLLAR
    MULTINATIONAL
    

    View Slide

67. AGILITY
    REASON
    #1:
    

    View Slide

68. AGILITY
    EARLY STAGE
    START-UP
    REASON
    #1:
    

    View Slide

69. AGILITY
    EARLY STAGE
    START-UP
    BILLION DOLLAR
    MULTINATIONAL
    REASON
    #1:
    

    View Slide

70. FLY WHEELS
    FOR AGILITY
    SECURITY
    SCALE
    PERFORMANCE
    RELIABILITY
    COST
    

    View Slide

71. ADOPTION
    PATTERNS OF How are enterprises
    using the cloud in
    2014?
    

    View Slide

72. DEVELOPMENT &
    TESTING
    SHAREPOINT & SAP
    ORACLE
    SAP
    

    View Slide

73. NEW WORKLOADS
    GLOBAL DEALS ENGINE
    PRODUCT PROTOTYPING
    MOBILE MUSIC DISCOVERY
    HOTEL BOOKING
    VIDEO STREAMING
    

    View Slide

74. SUPPLEMENT
    EXISTING
    WORKFLOWS
    operational applications
    equipment leasing
    analytics
    with the cloud
    

    View Slide

75. SUPPLEMENT
    EXISTING
    WORKFLOWS
    with on-premises infrastructure
    AWS serves up application
    content & data
    Integration back to Samsung
    Data Centers
    for financial transactions
    

    View Slide

76. MIGRATING
    EXISTING
    APPLICATIONS
    500 web properties in 5 months
    CLINICAL TRIALS
    1,700 web properties
    

    View Slide

77. ALL IN
    “WE HAVE TO BE GREAT AT A NUMBER OF THINGS...
    OPERATING DATA CENTERS IS NOT ONE OF THOSE THINGS”
    “WE ARE A HOSPITALITY MANAGEMENT COMPANY,
    NOT AN IT SERVICES COMPANY”
    “FRIENDS DON’T LET FRIENDS
    BUILD DATA CENTERS”
    INFOR CLOUDSUITE
    micro-vertical BUSINESS APPS
    

    View Slide

78. IN SUMMARY...
    DATA DRIVES INNOVATION WHEN
    COMBINED WITH UTILITY COMPUTING
    Broad and deep services
    enable new business
    AGILITY REMAINS THE #1 REASON
    FOR ADOPTING THE AWS CLOUD
    ENTERPRISES LEAD WITh
    DEV, TEST & NEW WORKLOADS
    MIGRATION AND SUPPLEMENTS
    QUICKLY FOLLOW
    Increasingly ‘ALL IN’
    

    View Slide

79. @mza
    MATTHEW@AMAZON.COM
    THANK YOU
    

    View Slide

80. More sessions from AWS today
    11.00am: Transforming Applications into SaaS
    11.40am: Big Data Is Best Served Fresh: Real-time Analytics in the Cloud
    1.55pm: What is the Internet of Things – and how can it help my business?
    4.00pm: Moving Enterprises to the Cloud the Right Way and for the Right Reasons
    4.40pm: Creating a Cloud Based DR Solution for your Enterprise
    

    View Slide