Unevenly Distributed

QCon London 2016 Keynote.
Includes a small number of bonus slides that I cut from the final presentation as delivered in order to keep within time.

Transcript

1. Unevenly
   Adrian Colyer
   @adriancolyer
   Distributed
   

   View Slide

2. blog.acolyer.org
   350
   Foundations
   Frontiers
   

   View Slide

3. Brain
   storm
   01
   02
   05
   04
   rainstorm
   03
   5 Reasons to <3 Papers
   Thinking
   tools
   Raise
   Expectations
   Applied
   Lessons The Great
   Conversation
   Uneven
   Distribution
   3
   

   View Slide

4. Frank McSherry
   Scalability - but at what CoST?
   4
   

   View Slide

5. View Slide

6. But you have BIG Data!
   6
   Zipf Distribution
   “Working sets are Zipf-
   distributed. We can
   therefore store in memory all
   but the very largest
   datasets.”
   

   View Slide

7. Musketeer
   7
   One for all?
   

   View Slide

8. Approx Hadoop
   8
   32x!
   

   View Slide

9. How to design DCFT Modules
   Design Patterns
   9
   Experience with Rules-Based Programming for Distributed Concurrent Fault-Tolerant Code -
   Stutsman et al. 2015
   

   View Slide

10. Improve your API Design
    The Scalable Commutativity Rule
    10
    

    View Slide

11. Thinking about the System
    11
    ?
    Memories, Guesses,
    Apologies
    &
    

    View Slide

12. Raising Your
    Expectations
    12
    

    View Slide

13. TLS
    13
    54 CVEs
    Jan ‘14 - Jan ‘15
    ! Error prone languages
    ! Lack of Separation
    ! Ambiguous and
    Untestable Spec
    Surely we can do
    better?
    

    View Slide

14. What if it just worked first time?
    Iron Fleet
    14
    High Level Spec
    (State Machine)
    Abstract
    Distributed
    Protocol
    Protocol
    Implementation
    

    View Slide

15. Do Less Testing!
    15
    Relative Improvement Cost Improvement
    Test Executions 40.58%
    Test Time 40.31% $1,567,608
    Test Result Inspection 33.04% $61,533
    Escaped Defects 0.20% ($11,971)
    Total Cost Balance $1,617,170
    Microsoft Windows 8.1
    

    View Slide

16. View Slide

17. Lessons from the
    Field
    17
    

    View Slide

18. at Facebook
    A Masterclass in Config Mgt
    18
    

    View Slide

19. lessons from Google
    Machine Learning Systems
    19
    Feature
    Management
    Visualisation
    Relative Metrics
    Systematic Bias
    Correction
    Alerts on action
    Thresholds
    01
    02
    03
    04
    05
    

    View Slide

20. And the Syntopicon
    The Great Conversation
    20
    

    View Slide

21. Robotics
    Security
    Distributed
    Systems
    Databases
    Machine Learning
    Programming
    Languages
    Broad Exposure to Problems and their Solutions
    Cross-Fertilization
    And Many More
    Operating Systems, Algorithms,
    Networking,Optimisation, SW Engineering,...
    21
    

    View Slide

22. TPC-C - 1992
    22
    

    View Slide

23. TPC-C Published Record Holder
    23
    Mar 26th 2013
    Date
    Oracle 11g r2 Enterprise Edition w. Partitioning
    Database Manager
    8,552,523 (8.5M)
    Performance (tpmC)
    142,542 (143K)
    Performance (tps)
    $4,663,073
    System Cost
    8
    #Processors
    128
    #Cores
    1024
    #Threads
    

    View Slide

24. and I-Confluence Analysis
    Coordination Avoidance
    24
    TPC-C
    

    View Slide

25. Multi-Partition Transactions at Scale
    25
    

    View Slide

26. Turning your world Upside Down
    Unevenly Distributed
    

    View Slide

27. Human
    computers
    at Dryden by NACA (NASA) -
    Dryden Flight Research Center
    Photo Collection
    http://www.dfrc.nasa.
    gov/Gallery/Photo/Places/HTML/E49-54.html.
    Licensed under Public Domain via Commons -
    https://commons.wikimedia.org/wiki/File:
    Human_computers_-_Dryden.jpg#/media/File:
    Human_computers_-_Dryden.jpg
    

    View Slide

28. Computing on a Human Scale
    28
    10ns
    70ns
    10ms
    10s
    1:10s
    116d
    Registers
    & L1-L3
    File on
    desk
    Main
    memory
    Office filing
    cabinet
    HDD
    Trip to the
    warehouse
    

    View Slide

29. Compute
    HTM
    Persistent Memory NI
    FPGA
    GPUs
    Memory
    NVDIMMs
    Persistent Memory
    Networking
    100GbE
    RDMA
    Storage
    NVMe
    Next-gen NVM
    Next Generation Hardware
    All Change Please
    29
    

    View Slide

30. View Slide

31. 2-10m
    Computing on a Human Scale
    31
    10s
    1:10s
    116d
    File on
    desk
    Office filing
    cabinet
    Trip to the
    warehouse
    4x capacity
    fireproof local
    filing cabinets
    23-40m
    Phone
    another office
    (RDMA)
    3h20m Next-gen
    warehouse
    

    View Slide

32. The New ~Numbers Everyone Should Know
    32
    Latency Bandwidth Capacity/IOPS
    Register 0.25ns
    L1 cache 1ns
    L2 cache 3ns 8MB
    L3 cache 11ns 45MB
    DRAM 62ns 120GBs 6TB - 4 socket
    NVRAM’ DIMM 620ns 60GBs 24TB - 4 socket
    1-sided RDMA in Data Center 1.4us 100GbE ~700K IOPS
    RPC in Data Center 2.4us 100GbE ~400K IOPS
    NVRAM’ NVMe 12us 6GBs 16TB/disk,~2M/600K
    NVRAM’ NVMf 90us 5GBs 16TB/disk, ~700/600K
    

    View Slide

33. Low Latency - RAMCloud
    33
    Reads
    5μs
    Writes
    13.5μs
    Transactions
    20μs
    5-object Txns
    27μs
    TPC-C (10 nodes)
    35K tps
    

    View Slide

34. No Compromises - FaRM
    34
    TPC-C (90 nodes)
    4.5M tps
    99%ile
    1.9ms
    KV (per node)
    6.3M qps
    at peak throughput
    41μs
    

    View Slide

35. No Compromises
    35
    “This paper demonstrates that new software in modern
    data centers can eliminate the need to compromise. It
    describes the transaction, replication, and recovery
    protocols in FaRM, a main memory distributed computing
    platform. FaRM provides distributed ACID transactions
    with strict serializability, high availability, high
    throughput and low latency. These protocols were
    designed from first principles to leverage two hardware
    trends appearing in data centers: fast commodity
    networks with RDMA and an inexpensive approach to
    providing non-volatile DRAM.”
    

    View Slide

36. DrTM
    The Doctor will see you now
    36
    5.5M tps on TPC-C
    6-node cluster.
    

    View Slide

37. Some things Change, Some stay the Same
    37
    

    View Slide

38. A Brave New World
    38
    Fast RDMA networks +
    Ample Persistent Memory +
    Hardware Transactions +
    Enhanced HW Cache Management +
    Super-fast Storage +
    On-board FPGAs + GPUs + … = ???
    

    View Slide

39. Brain
    storm
    01
    02
    05
    04
    rainstorm
    03
    5 Reasons to <3 Papers
    Thinking
    tools
    Raise
    Expectations
    Applied
    Lessons The Great
    Conversation
    Uneven
    Distribution
    39
    

    View Slide

40. A new paper every weekday
    Published at http://blog.acolyer.org.
    01
    Delivered Straight to your inbox
    If you prefer email-based subscription to read at
    your leisure.
    02
    Announced on Twitter
    I’m @adriancolyer.
    03
    Go to a Papers We Love Meetup
    A repository of academic computer science papers
    and a community who loves reading them.
    04
    Share what you learn
    Anyone can take part in the great conversation.
    05
    

    View Slide

41. THANK YOU !
    @adriancolyer
    

    View Slide