Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unevenly Distributed

Unevenly Distributed

QCon London 2016 Keynote.
Includes a small number of bonus slides that I cut from the final presentation as delivered in order to keep within time.

Adrian Colyer

March 07, 2016

More Decks by Adrian Colyer

Other Decks in Technology


  1. Unevenly Adrian Colyer @adriancolyer Distributed

  2. blog.acolyer.org 350 Foundations Frontiers

  3. Brain storm 01 02 05 04 rainstorm 03 5 Reasons

    to <3 Papers Thinking tools Raise Expectations Applied Lessons The Great Conversation Uneven Distribution 3
  4. Frank McSherry Scalability - but at what CoST? 4

  5. 5

  6. But you have BIG Data! 6 Zipf Distribution “Working sets

    are Zipf- distributed. We can therefore store in memory all but the very largest datasets.”
  7. Musketeer 7 One for all?

  8. Approx Hadoop 8 32x!

  9. How to design DCFT Modules Design Patterns 9 Experience with

    Rules-Based Programming for Distributed Concurrent Fault-Tolerant Code - Stutsman et al. 2015
  10. Improve your API Design The Scalable Commutativity Rule 10

  11. Thinking about the System 11 ? Memories, Guesses, Apologies &

  12. Raising Your Expectations 12

  13. TLS 13 54 CVEs Jan ‘14 - Jan ‘15 !

    Error prone languages ! Lack of Separation ! Ambiguous and Untestable Spec Surely we can do better?
  14. What if it just worked first time? Iron Fleet 14

    High Level Spec (State Machine) Abstract Distributed Protocol Protocol Implementation
  15. Do Less Testing! 15 Relative Improvement Cost Improvement Test Executions

    40.58% Test Time 40.31% $1,567,608 Test Result Inspection 33.04% $61,533 Escaped Defects 0.20% ($11,971) Total Cost Balance $1,617,170 Microsoft Windows 8.1
  16. 16

  17. Lessons from the Field 17

  18. at Facebook A Masterclass in Config Mgt 18

  19. lessons from Google Machine Learning Systems 19 Feature Management Visualisation

    Relative Metrics Systematic Bias Correction Alerts on action Thresholds 01 02 03 04 05
  20. And the Syntopicon The Great Conversation 20

  21. Robotics Security Distributed Systems Databases Machine Learning Programming Languages Broad

    Exposure to Problems and their Solutions Cross-Fertilization And Many More Operating Systems, Algorithms, Networking,Optimisation, SW Engineering,... 21
  22. TPC-C - 1992 22

  23. TPC-C Published Record Holder 23 Mar 26th 2013 Date Oracle

    11g r2 Enterprise Edition w. Partitioning Database Manager 8,552,523 (8.5M) Performance (tpmC) 142,542 (143K) Performance (tps) $4,663,073 System Cost 8 #Processors 128 #Cores 1024 #Threads
  24. and I-Confluence Analysis Coordination Avoidance 24 TPC-C

  25. Multi-Partition Transactions at Scale 25

  26. Turning your world Upside Down Unevenly Distributed

  27. Human computers at Dryden by NACA (NASA) - Dryden Flight

    Research Center Photo Collection http://www.dfrc.nasa. gov/Gallery/Photo/Places/HTML/E49-54.html. Licensed under Public Domain via Commons - https://commons.wikimedia.org/wiki/File: Human_computers_-_Dryden.jpg#/media/File: Human_computers_-_Dryden.jpg
  28. Computing on a Human Scale 28 10ns 70ns 10ms 10s

    1:10s 116d Registers & L1-L3 File on desk Main memory Office filing cabinet HDD Trip to the warehouse
  29. Compute HTM Persistent Memory NI FPGA GPUs Memory NVDIMMs Persistent

    Memory Networking 100GbE RDMA Storage NVMe Next-gen NVM Next Generation Hardware All Change Please 29
  30. 30

  31. 2-10m Computing on a Human Scale 31 10s 1:10s 116d

    File on desk Office filing cabinet Trip to the warehouse 4x capacity fireproof local filing cabinets 23-40m Phone another office (RDMA) 3h20m Next-gen warehouse
  32. The New ~Numbers Everyone Should Know 32 Latency Bandwidth Capacity/IOPS

    Register 0.25ns L1 cache 1ns L2 cache 3ns 8MB L3 cache 11ns 45MB DRAM 62ns 120GBs 6TB - 4 socket NVRAM’ DIMM 620ns 60GBs 24TB - 4 socket 1-sided RDMA in Data Center 1.4us 100GbE ~700K IOPS RPC in Data Center 2.4us 100GbE ~400K IOPS NVRAM’ NVMe 12us 6GBs 16TB/disk,~2M/600K NVRAM’ NVMf 90us 5GBs 16TB/disk, ~700/600K
  33. Low Latency - RAMCloud 33 Reads 5μs Writes 13.5μs Transactions

    20μs 5-object Txns 27μs TPC-C (10 nodes) 35K tps
  34. No Compromises - FaRM 34 TPC-C (90 nodes) 4.5M tps

    99%ile 1.9ms KV (per node) 6.3M qps at peak throughput 41μs
  35. No Compromises 35 “This paper demonstrates that new software in

    modern data centers can eliminate the need to compromise. It describes the transaction, replication, and recovery protocols in FaRM, a main memory distributed computing platform. FaRM provides distributed ACID transactions with strict serializability, high availability, high throughput and low latency. These protocols were designed from first principles to leverage two hardware trends appearing in data centers: fast commodity networks with RDMA and an inexpensive approach to providing non-volatile DRAM.”
  36. DrTM The Doctor will see you now 36 5.5M tps

    on TPC-C 6-node cluster.
  37. Some things Change, Some stay the Same 37

  38. A Brave New World 38 Fast RDMA networks + Ample

    Persistent Memory + Hardware Transactions + Enhanced HW Cache Management + Super-fast Storage + On-board FPGAs + GPUs + … = ???
  39. Brain storm 01 02 05 04 rainstorm 03 5 Reasons

    to <3 Papers Thinking tools Raise Expectations Applied Lessons The Great Conversation Uneven Distribution 39
  40. A new paper every weekday Published at http://blog.acolyer.org. 01 Delivered

    Straight to your inbox If you prefer email-based subscription to read at your leisure. 02 Announced on Twitter I’m @adriancolyer. 03 Go to a Papers We Love Meetup A repository of academic computer science papers and a community who loves reading them. 04 Share what you learn Anyone can take part in the great conversation. 05
  41. THANK YOU ! @adriancolyer