Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unevenly Distributed

Unevenly Distributed

QCon London 2016 Keynote.
Includes a small number of bonus slides that I cut from the final presentation as delivered in order to keep within time.

Adrian Colyer

March 07, 2016
Tweet

More Decks by Adrian Colyer

Other Decks in Technology

Transcript

  1. Unevenly
    Adrian Colyer
    @adriancolyer
    Distributed

    View Slide

  2. blog.acolyer.org
    350
    Foundations
    Frontiers

    View Slide

  3. Brain
    storm
    01
    02
    05
    04
    rainstorm
    03
    5 Reasons to <3 Papers
    Thinking
    tools
    Raise
    Expectations
    Applied
    Lessons The Great
    Conversation
    Uneven
    Distribution
    3

    View Slide

  4. Frank McSherry
    Scalability - but at what CoST?
    4

    View Slide

  5. 5

    View Slide

  6. But you have BIG Data!
    6
    Zipf Distribution
    “Working sets are Zipf-
    distributed. We can
    therefore store in memory all
    but the very largest
    datasets.”

    View Slide

  7. Musketeer
    7
    One for all?

    View Slide

  8. Approx Hadoop
    8
    32x!

    View Slide

  9. How to design DCFT Modules
    Design Patterns
    9
    Experience with Rules-Based Programming for Distributed Concurrent Fault-Tolerant Code -
    Stutsman et al. 2015

    View Slide

  10. Improve your API Design
    The Scalable Commutativity Rule
    10

    View Slide

  11. Thinking about the System
    11
    ?
    Memories, Guesses,
    Apologies
    &

    View Slide

  12. Raising Your
    Expectations
    12

    View Slide

  13. TLS
    13
    54 CVEs
    Jan ‘14 - Jan ‘15
    ! Error prone languages
    ! Lack of Separation
    ! Ambiguous and
    Untestable Spec
    Surely we can do
    better?

    View Slide

  14. What if it just worked first time?
    Iron Fleet
    14
    High Level Spec
    (State Machine)
    Abstract
    Distributed
    Protocol
    Protocol
    Implementation

    View Slide

  15. Do Less Testing!
    15
    Relative Improvement Cost Improvement
    Test Executions 40.58%
    Test Time 40.31% $1,567,608
    Test Result Inspection 33.04% $61,533
    Escaped Defects 0.20% ($11,971)
    Total Cost Balance $1,617,170
    Microsoft Windows 8.1

    View Slide

  16. 16

    View Slide

  17. Lessons from the
    Field
    17

    View Slide

  18. at Facebook
    A Masterclass in Config Mgt
    18

    View Slide

  19. lessons from Google
    Machine Learning Systems
    19
    Feature
    Management
    Visualisation
    Relative Metrics
    Systematic Bias
    Correction
    Alerts on action
    Thresholds
    01
    02
    03
    04
    05

    View Slide

  20. And the Syntopicon
    The Great Conversation
    20

    View Slide

  21. Robotics
    Security
    Distributed
    Systems
    Databases
    Machine Learning
    Programming
    Languages
    Broad Exposure to Problems and their Solutions
    Cross-Fertilization
    And Many More
    Operating Systems, Algorithms,
    Networking,Optimisation, SW Engineering,...
    21

    View Slide

  22. TPC-C - 1992
    22

    View Slide

  23. TPC-C Published Record Holder
    23
    Mar 26th 2013
    Date
    Oracle 11g r2 Enterprise Edition w. Partitioning
    Database Manager
    8,552,523 (8.5M)
    Performance (tpmC)
    142,542 (143K)
    Performance (tps)
    $4,663,073
    System Cost
    8
    #Processors
    128
    #Cores
    1024
    #Threads

    View Slide

  24. and I-Confluence Analysis
    Coordination Avoidance
    24
    TPC-C

    View Slide

  25. Multi-Partition Transactions at Scale
    25

    View Slide

  26. Turning your world Upside Down
    Unevenly Distributed

    View Slide

  27. Human
    computers
    at Dryden by NACA (NASA) -
    Dryden Flight Research Center
    Photo Collection
    http://www.dfrc.nasa.
    gov/Gallery/Photo/Places/HTML/E49-54.html.
    Licensed under Public Domain via Commons -
    https://commons.wikimedia.org/wiki/File:
    Human_computers_-_Dryden.jpg#/media/File:
    Human_computers_-_Dryden.jpg

    View Slide

  28. Computing on a Human Scale
    28
    10ns
    70ns
    10ms
    10s
    1:10s
    116d
    Registers
    & L1-L3
    File on
    desk
    Main
    memory
    Office filing
    cabinet
    HDD
    Trip to the
    warehouse

    View Slide

  29. Compute
    HTM
    Persistent Memory NI
    FPGA
    GPUs
    Memory
    NVDIMMs
    Persistent Memory
    Networking
    100GbE
    RDMA
    Storage
    NVMe
    Next-gen NVM
    Next Generation Hardware
    All Change Please
    29

    View Slide

  30. 30

    View Slide

  31. 2-10m
    Computing on a Human Scale
    31
    10s
    1:10s
    116d
    File on
    desk
    Office filing
    cabinet
    Trip to the
    warehouse
    4x capacity
    fireproof local
    filing cabinets
    23-40m
    Phone
    another office
    (RDMA)
    3h20m Next-gen
    warehouse

    View Slide

  32. The New ~Numbers Everyone Should Know
    32
    Latency Bandwidth Capacity/IOPS
    Register 0.25ns
    L1 cache 1ns
    L2 cache 3ns 8MB
    L3 cache 11ns 45MB
    DRAM 62ns 120GBs 6TB - 4 socket
    NVRAM’ DIMM 620ns 60GBs 24TB - 4 socket
    1-sided RDMA in Data Center 1.4us 100GbE ~700K IOPS
    RPC in Data Center 2.4us 100GbE ~400K IOPS
    NVRAM’ NVMe 12us 6GBs 16TB/disk,~2M/600K
    NVRAM’ NVMf 90us 5GBs 16TB/disk, ~700/600K

    View Slide

  33. Low Latency - RAMCloud
    33
    Reads
    5μs
    Writes
    13.5μs
    Transactions
    20μs
    5-object Txns
    27μs
    TPC-C (10 nodes)
    35K tps

    View Slide

  34. No Compromises - FaRM
    34
    TPC-C (90 nodes)
    4.5M tps
    99%ile
    1.9ms
    KV (per node)
    6.3M qps
    at peak throughput
    41μs

    View Slide

  35. No Compromises
    35
    “This paper demonstrates that new software in modern
    data centers can eliminate the need to compromise. It
    describes the transaction, replication, and recovery
    protocols in FaRM, a main memory distributed computing
    platform. FaRM provides distributed ACID transactions
    with strict serializability, high availability, high
    throughput and low latency. These protocols were
    designed from first principles to leverage two hardware
    trends appearing in data centers: fast commodity
    networks with RDMA and an inexpensive approach to
    providing non-volatile DRAM.”

    View Slide

  36. DrTM
    The Doctor will see you now
    36
    5.5M tps on TPC-C
    6-node cluster.

    View Slide

  37. Some things Change, Some stay the Same
    37

    View Slide

  38. A Brave New World
    38
    Fast RDMA networks +
    Ample Persistent Memory +
    Hardware Transactions +
    Enhanced HW Cache Management +
    Super-fast Storage +
    On-board FPGAs + GPUs + … = ???

    View Slide

  39. Brain
    storm
    01
    02
    05
    04
    rainstorm
    03
    5 Reasons to <3 Papers
    Thinking
    tools
    Raise
    Expectations
    Applied
    Lessons The Great
    Conversation
    Uneven
    Distribution
    39

    View Slide

  40. A new paper every weekday
    Published at http://blog.acolyer.org.
    01
    Delivered Straight to your inbox
    If you prefer email-based subscription to read at
    your leisure.
    02
    Announced on Twitter
    I’m @adriancolyer.
    03
    Go to a Papers We Love Meetup
    A repository of academic computer science papers
    and a community who loves reading them.
    04
    Share what you learn
    Anyone can take part in the great conversation.
    05

    View Slide

  41. THANK YOU !
    @adriancolyer

    View Slide