Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Field Notes from Expeditions in the Cloud

Field Notes from Expeditions in the Cloud

Keynote from Cloud Expo 2014, in NYC.

Matt Wood

June 10, 2014
Tweet

More Decks by Matt Wood

Other Decks in Technology

Transcript

  1. from
    Dr. Matt Wood
    [email protected]
    General Manager, Data Science
    a presentation by
    Matt Wood
    N
    E
    S
    W
    @mza
    FIELD NOTES
    EXPEDITIONS
    IN THE CLOUD

    View Slide

  2. RETAIL
    280

    View Slide

  3. RETAIL MERCH
    280

    View Slide

  4. RETAIL MERCH AWS
    280

    View Slide

  5. 14th March 2006 S3
    25th AUGUST 2006 EC2
    22nd OCTOBER 2009 RDS
    15th FEBRUARY 2013 Redshift
    18th JANUARY 2012 DYNAMODB
    RETAIL MERCH AWS
    280

    View Slide

  6. 14th March 2006 S3
    25th AUGUST 2006 EC2
    22nd OCTOBER 2009 RDS
    15th FEBRUARY 2013 Redshift
    18th JANUARY 2012 DYNAMODB
    RETAIL MERCH AWS 2008
    2009
    2010
    2011
    2012
    2013
    159
    82
    61
    48
    24
    280

    View Slide

  7. 14th March 2006 S3
    25th AUGUST 2006 EC2
    22nd OCTOBER 2009 RDS
    15th FEBRUARY 2013 Redshift 170 launches in 2014
    18th JANUARY 2012 DYNAMODB
    RETAIL MERCH AWS 2008
    2009
    2010
    2011
    2012
    2013
    159
    82
    61
    48
    24
    280

    View Slide

  8. LOTS OF DATA

    View Slide

  9. LOTS OF DATA
    LOTS OF USES

    View Slide

  10. LOTS OF DATA
    LOTS OF USES
    LOTS OF USERS

    View Slide

  11. LOTS OF DATA
    LOTS OF USES
    LOTS OF USERS
    LOTS OF LOCATIONS

    View Slide

  12. LOTS OF DATA
    LOTS OF USES
    LOTS OF USERS
    LOTS OF LOCATIONS
    COMPLEXITY

    View Slide

  13. SOLAR PANELS
    Notes on
    CAN we make A
    better solar panel?”

    View Slide

  14. SOLAR PANELS
    Notes on
    CAN we make A
    better solar panel?”

    Synthesize
    PURIFY
    ANALYZE
    PHOTO CELL?

    View Slide

  15. SOLAR PANELS
    Notes on
    CAN we make A
    better solar panel?”

    Synthesize
    PURIFY
    ANALYZE
    PHOTO CELL?
    GRAD
    STUDENT
    EQUIPMENT
    +

    View Slide

  16. SOLAR PANELS
    Notes on
    CAN we make A
    better solar panel?”

    205,000
    organic molecules
    Synthesize
    PURIFY
    ANALYZE
    PHOTO CELL?
    GRAD
    STUDENT
    EQUIPMENT
    +

    View Slide

  17. View Slide

  18. 1.21 PFLOPS
    16,788 instances

    View Slide

  19. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours

    View Slide

  20. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k

    View Slide

  21. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k
    COST MODELS
    ENABLE INNOVATION
    UTILIZATION

    View Slide

  22. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k
    COST MODELS
    ENABLE INNOVATION
    UTILIZATION
    RESERVED CAPACITY

    View Slide

  23. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k
    COST MODELS
    ENABLE INNOVATION
    UTILIZATION
    RESERVED CAPACITY
    ON DEMAND

    View Slide

  24. 1.21 PFLOPS
    16,788 instances
    264 years of compute time
    <18 hours
    $33k
    COST MODELS
    ENABLE INNOVATION
    UTILIZATION
    RESERVED CAPACITY
    ON DEMAND

    View Slide

  25. VIDEO GAMES
    Notes on

    View Slide

  26. VIDEO GAMES
    Notes on
    “WHO BUYS VIDEO GAMES?”

    View Slide

  27. VIDEO GAMES
    Notes on
    “WHO BUYS VIDEO GAMES?”
    3.5 billion records
    13 TB of click stream logs
    71 million unique cookies
    C
    luster

    View Slide

  28. VIDEO GAMES
    Notes on
    “WHO BUYS VIDEO GAMES?”
    3.5 billion records
    13 TB of click stream logs
    71 million unique cookies
    C
    luster
    HUGE TVs

    View Slide

  29. 500% return on
    AD SPEND
    17000% decrease in
    procurement time

    View Slide

  30. RETARGETING
    Notes on

    View Slide

  31. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”

    View Slide

  32. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”

    AD SERVERS

    View Slide

  33. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”

    AD SERVERS
    BIDDING ENGINE

    View Slide

  34. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”

    AD SERVERS
    BIDDING ENGINE
    DYNAMODB

    View Slide

  35. RETARGETING
    Notes on HOW DO WE SERVE THE
    BEST POSSIBLE AD?”

    AD SERVERS
    BIDDING ENGINE
    10ms
    20MS
    20MS 20MS
    HARD LIMIT: 100ms
    10k advertisers
    100 countries
    1.5PB in S3
    DYNAMODB

    View Slide

  36. 100 countries

    View Slide

  37. 100 countries

    View Slide

  38. 100 countries
    HUNDREDS OF MILLIONS
    OF RECORDS
    10ks WRITES PER SECOND
    in sync at
    WITH APACHE STORM

    View Slide

  39. DYNAMODB THROUGHPUT
    AVERAGE LATENCY
    11/1/2013 11/7/2013

    View Slide

  40. DYNAMODB THROUGHPUT
    AVERAGE LATENCY
    # REQUESTS
    11/1/2013 11/7/2013

    View Slide

  41. DYNAMODB THROUGHPUT
    AVERAGE LATENCY
    # REQUESTS
    “THE LAW OF LARGE
    NUMBERS IS YOUR FRIEND
    UNTIL YOU REACH
    LARGE NUMBERS”
    11/1/2013 11/7/2013

    View Slide

  42. DYNAMODB THROUGHPUT
    “THE LAW OF LARGE
    NUMBERS IS YOUR FRIEND
    UNTIL YOU REACH
    LARGE NUMBERS”
    PRECISE DESCRIPTION
    FORMAL METHODS
    TEST, TEST, TEST
    AVERAGE LATENCY
    # REQUESTS
    11/1/2013 11/7/2013

    View Slide

  43. SOCIAL GAMINg
    Notes on

    View Slide

  44. SOCIAL GAMINg
    Notes on
    CAN PHYSICAL AND VIRTUAL
    WORLDS INTERACT?”

    SOCIAL EXPERIENCE
    BUILT RIGHT INTO THE GAMES

    View Slide

  45. SOCIAL GAMINg
    Notes on
    CAN PHYSICAL AND VIRTUAL
    WORLDS INTERACT?”

    SOCIAL EXPERIENCE
    BUILT RIGHT INTO THE GAMES
    LOW LATENCY
    HIGH AVAILABILITY
    ANALYTICS

    View Slide

  46. JAPAN
    US EUROPE

    View Slide

  47. JAPAN
    US EUROPE
    S3 CLOUDFRONT
    VPC

    View Slide

  48. JAPAN
    US EUROPE
    S3 CLOUDFRONT
    VPC
    EC2 EC2 EC2
    AZ1 AZ2 AZ3

    View Slide

  49. JAPAN
    US EUROPE
    REDSHIFT
    S3 CLOUDFRONT
    VPC
    EC2 EC2 EC2
    AZ1 AZ2 AZ3

    View Slide

  50. GENOMICS
    Notes on

    View Slide

  51. Chromosome 11 GENOMICS
    Notes on

    View Slide

  52. Chromosome 11
    Chromosome X
    GENOMICS
    Notes on

    View Slide

  53. Chromosome 11
    Chromosome X
    GENOMICS
    Notes on
    Chromosome 19

    View Slide

  54. Chromosome 11
    Chromosome X
    Chromosome 15
    GENOMICS
    Notes on
    Chromosome 19

    View Slide

  55. Chromosome 11
    Chromosome X
    Chromosome 15
    GENOMICS
    Notes on
    Chromosome 19

    View Slide

  56. Chromosome 11
    Chromosome X
    Chromosome 15
    GENOMICS
    Notes on
    Chromosome 19

    View Slide

  57. DNA SEQUENCING COSTS
    < $1000
    PER GENOME

    View Slide

  58. DNA SEQUENCING COSTS
    < $1000
    PER GENOME

    View Slide

  59. DNA SEQUENCING COSTS
    < $1000
    PER GENOME
    S3
    COLLABORATION and TOOLING

    View Slide

  60. View Slide

  61. CLINICAL TRIALs
    Notes on

    View Slide

  62. CLINICAL TRIALs
    Notes on
    BUSINESS DRIVERS
    Speeding server provisioning for R&D apps
    Extending capacity for internal grid
    environments
    Slowing internally hosted compute
    infrastructure growth
    On-boarding security, validation services and
    compliance
    Reducing cost while extending
    capabilities

    View Slide

  63. CLINICAL TRIAL SIMULATIONS
    Individual Clinical Trial Simulation Run Time (Min)
    Total number of trials
    Number of servers
    Number of CPUS
    TOTAL ANALYSIS RUN TIME (hours)
    COST
    INTERNAL AWS
    56 56
    2000 2000
    2
    32
    60
    ??? $336
    1.2
    2048
    256

    View Slide

  64. REDUCED BURDEN ON
    pEDIATRIC SUBJECTS
    NUmber of subjects
    NUmber of blood samples per subject
    length of stay per subject
    length of study
    total study cost
    traditional
    DESIGN
    DESIGN OPTIMIZED
    USING CLINICAL
    TRIAL SIMULATION
    60 40
    12 5
    72 hours
    2.5 years
    $700k $250k
    1.7 years
    26 hours

    View Slide

  65. CONNECTED
    DEVICES
    GENOMICS
    CLINICAL
    TRIALS
    GAMING
    MOBILE
    DEVELOPMENT
    ADVERTISING
    HOSPITALITY
    RETAIL
    E-COMMERCE MANUFACTURING
    TRAVEL
    FREE STEAK
    SOCIAL APPS
    FINANCIAL
    ANALYTICS
    MEDIA STREAMING
    DISASTER
    RECOVERY
    ARCHIVAL
    INTERACTIVE
    TV APPS
    WEB APPS
    MARS
    EXPLORATION

    View Slide

  66. WHERE TO
    FOCUS?
    EARLY STAGE
    START-UP
    BILLION DOLLAR
    MULTINATIONAL

    View Slide

  67. AGILITY
    REASON
    #1:

    View Slide

  68. AGILITY
    EARLY STAGE
    START-UP
    REASON
    #1:

    View Slide

  69. AGILITY
    EARLY STAGE
    START-UP
    BILLION DOLLAR
    MULTINATIONAL
    REASON
    #1:

    View Slide

  70. FLY WHEELS
    FOR AGILITY
    SECURITY
    SCALE
    PERFORMANCE
    RELIABILITY
    COST

    View Slide

  71. ADOPTION
    PATTERNS OF How are enterprises
    using the cloud in
    2014?

    View Slide

  72. DEVELOPMENT &
    TESTING
    SHAREPOINT & SAP
    ORACLE
    SAP

    View Slide

  73. NEW WORKLOADS
    GLOBAL DEALS ENGINE
    PRODUCT PROTOTYPING
    MOBILE MUSIC DISCOVERY
    HOTEL BOOKING
    VIDEO STREAMING

    View Slide

  74. SUPPLEMENT
    EXISTING
    WORKFLOWS
    operational applications
    equipment leasing
    analytics
    with the cloud

    View Slide

  75. SUPPLEMENT
    EXISTING
    WORKFLOWS
    with on-premises infrastructure
    AWS serves up application
    content & data
    Integration back to Samsung
    Data Centers
    for financial transactions

    View Slide

  76. MIGRATING
    EXISTING
    APPLICATIONS
    500 web properties in 5 months
    CLINICAL TRIALS
    1,700 web properties

    View Slide

  77. ALL IN
    “WE HAVE TO BE GREAT AT A NUMBER OF THINGS...
    OPERATING DATA CENTERS IS NOT ONE OF THOSE THINGS”
    “WE ARE A HOSPITALITY MANAGEMENT COMPANY,
    NOT AN IT SERVICES COMPANY”
    “FRIENDS DON’T LET FRIENDS
    BUILD DATA CENTERS”
    INFOR CLOUDSUITE
    micro-vertical BUSINESS APPS

    View Slide

  78. IN SUMMARY...
    DATA DRIVES INNOVATION WHEN
    COMBINED WITH UTILITY COMPUTING
    Broad and deep services
    enable new business
    AGILITY REMAINS THE #1 REASON
    FOR ADOPTING THE AWS CLOUD
    ENTERPRISES LEAD WITh
    DEV, TEST & NEW WORKLOADS
    MIGRATION AND SUPPLEMENTS
    QUICKLY FOLLOW
    Increasingly ‘ALL IN’

    View Slide

  79. @mza
    [email protected]
    THANK YOU

    View Slide

  80. More sessions from AWS today
    11.00am: Transforming Applications into SaaS
    11.40am: Big Data Is Best Served Fresh: Real-time Analytics in the Cloud
    1.55pm: What is the Internet of Things – and how can it help my business?
    4.00pm: Moving Enterprises to the Cloud the Right Way and for the Right Reasons
    4.40pm: Creating a Cloud Based DR Solution for your Enterprise

    View Slide