CraftConf 2016 - Building Scalable Stateful Services

CraftConf 2016 - Building Scalable Stateful Services

The Stateless Service design principle has become ubiquitous in the tech industry for creating horizontally scalable services. However our applications do have state, we just have moved all of it to caches and databases. Today as applications are becoming more data intensive and request latencies are expected to be incredibly low, we’d like the benefits of stateful services, like data locality and sticky consistency. In this talk I will address the benefits of stateful services, how to build them so that they scale, and discuss distributed and scalable services in the real world that implement these techniques successfully.

9128d500301ae51524e887bb680f471d?s=128

Caitie McCaffrey

April 29, 2016
Tweet

Transcript

  1. Building Scalable Stateful Services Craft Conf 2016

  2. Caitie McCaffrey Distributed Systems Engineer @caitie caitiem.com

  3. Stateless Services Service

  4. Stateless Services Service

  5. Stateless Services Service Service

  6. Stateless Services Service Service

  7. Stateless Services Service Service

  8. Stateless Services Service Service Service

  9. Stateless Services Service Service Service

  10. Stateless Services Service Service Service

  11. Stateless Services Service Service Service Service

  12. Stateless Services Service Service Service Service

  13. Stateless Services Service Service Service Service

  14. Stateless Services Service Service Service Service Service

  15. Stateless Services Service Service Service Service Service

  16. Data Shipping Paradigm Service Service Service

  17. Data Shipping Paradigm Service Service Service

  18. Data Shipping Paradigm Service Service Service

  19. Data Shipping Paradigm Service Service Service

  20. Data Shipping Paradigm Service Service Service

  21. Data Shipping Paradigm Service Service Service

  22. Data Shipping Paradigm Service Service Service

  23. Data Shipping Paradigm Service Service Service

  24. Data Shipping Paradigm Service Service Service

  25. Data Shipping Paradigm Service Service Service

  26. Data Shipping Paradigm Service Service Service

  27. Data Shipping Paradigm Service Service Service

  28. Data Shipping Paradigm Service Service Service

  29. Data Shipping Paradigm Service Service Service

  30. Data Shipping Paradigm Service Service Service

  31. Overview Benefits Building Real World Caution

  32. Data Locality For Low Latency & Data Intensive Services

  33. Function Shipping Paradigm Service Service Service

  34. Function Shipping Paradigm Service Service Service

  35. Function Shipping Paradigm Service Service Service

  36. Function Shipping Paradigm Service Service Service

  37. Function Shipping Paradigm Service Service Service

  38. Function Shipping Paradigm Service Service Service

  39. Function Shipping Paradigm Service Service Service

  40. Function Shipping Paradigm Service Service Service

  41. Function Shipping Paradigm Service Service Service

  42. Function Shipping Paradigm Service Service Service

  43. Function Shipping Paradigm Service Service Service

  44. Sticky Connections & Consistency Additional Available Consistency Models

  45. Sticky Connections

  46. Sticky Connections

  47. Linearizable Sequential Causal Pipelined Random Access Memory Read Your Write

    Monotonic Read Monotonic Write Write From Read Consistency Models CP Consistency AP Consistency
  48. Linearizable Sequential Causal Pipelined Random Access Memory Read Your Write

    Monotonic Read Monotonic Write Write From Read CP Consistency AP Consistency AP Consistency w/ Sticky Connections Consistency Models
  49. - Werner Vogel 2007 “Whether or not read-your-write, session and

    monotonic consistency can be achieved depends in general on the "stickiness" of clients to the server that executes the distributed protocol for them… Using sessions, which are sticky, makes this explicit and provides an exposure level that clients can reason about.”
  50. Building Sticky Connections For Low Latency & Data Intensive Services

  51. Building Sticky Connections

  52. Building Sticky Connections

  53. Persistent Connections Load Balancing Problems No Stickiness Once Connection Breaks

    Problems
  54. Persistent Connections Load Balancing Problems No Stickiness Once Connection Breaks

    Problems
  55. Routing Logic • Cluster Membership • Work Distribution Problems to

    Solve
  56. Routing Logic • Cluster Membership • Work Distribution Problems to

    Solve
  57. Routing Logic • Cluster Membership • Work Distribution Problems to

    Solve
  58. Cluster Membership Dynamic Static

  59. Static Cluster Membership Machine Outage means Partial Service Outage Downtime

    to Add Capacity
  60. Static Cluster Membership Machine Outage means Partial Service Outage Downtime

    to Add Capacity
  61. Static Cluster Membership Machine Outage means Partial Service Outage Downtime

    to Add Capacity
  62. Dynamic Cluster Membership Fault Tolerance

  63. Dynamic Cluster Membership Fault Tolerance

  64. Dynamic Cluster Membership Fault Tolerance

  65. Dynamic Cluster Membership Fault Tolerance

  66. Dynamic Cluster Membership Fault Tolerance

  67. Dynamic Cluster Membership Gossip Protocols Consensus Systems Availability vs Consistency

  68. Consensus Systems for Consistency Paxos ETCD Zookeeper Raft

  69. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  70. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  71. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  72. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  73. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  74. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  75. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  76. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  77. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  78. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  79. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  80. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  81. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  82. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  83. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  84. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  85. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  86. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  87. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  88. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  89. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information

    Dissemination • Pairwise Communication
  90. Work Distribution Consistent Hashing Distributed Hash Tables Random Placement

  91. Random Placement Write Anywhere Read from Everywhere

  92. Random Placement Write Anywhere Read from Everywhere

  93. Random Placement Write Anywhere Read from Everywhere

  94. Random Placement Write Anywhere Read from Everywhere

  95. Random Placement Write Anywhere Read from Everywhere

  96. Random Placement Write Anywhere Read from Everywhere

  97. Random Placement Write Anywhere Read from Everywhere

  98. Consistent Hashing Deterministic Placement Node A Node B Node C

    NodeD Consistent Hashing & Random Trees: Distributed caching protocols for relieving hot spots on the World Wide Web
  99. Distributed Hash Table Non- Deterministic Placement Node B Node A

    Node C
  100. Distributed Hash Table Non- Deterministic Placement Node B Node A

    Node C
  101. Distributed Hash Table Non- Deterministic Placement Node B Node A

    Node C
  102. Distributed Hash Table Non- Deterministic Placement Node B Node A

    Node C
  103. Distributed Hash Table Non- Deterministic Placement Node B Node A

    Node C
  104. Stateful Services In the Real World

  105. Scuba is a fast, scalable, distributed, in-memory database built at

    Facebook. It is the workhorse behind code regression analysis & bug report, revenue, and performance debugging Fan-out request to all machines in the cluster Compose Results Return Results and Completeness
  106. Scuba is a fast, scalable, distributed, in-memory database built at

    Facebook. It is the workhorse behind code regression analysis & bug report, revenue, and performance debugging Fan-out request to all machines in the cluster Compose Results Return Results and Completeness
  107. Scuba is a fast, scalable, distributed, in-memory database built at

    Facebook. It is the workhorse behind code regression analysis & bug report, revenue, and performance debugging Fan-out request to all machines in the cluster Compose Results Return Results and Completeness
  108. Twitter Nuthatch is a temporal set indexing service. Helps resolve

    glob queries in metric names. Static Cluster Membership + Hashing
  109. Twitter Nuthatch is a temporal set indexing service. Helps resolve

    glob queries in metric names. Static Cluster Membership + Hashing
  110. Twitter Nuthatch is a temporal set indexing service. Helps resolve

    glob queries in metric names. Static Cluster Membership + Hashing
  111. Uber Ringpop is an open- source Node.js library that brings

    application-layer sharding to many of their dispatching platform services. Swim Gossip Protocol Consistent Hashing +
  112. Uber Ringpop is an open- source Node.js library that brings

    application-layer sharding to many of their dispatching platform services. Swim Gossip Protocol Consistent Hashing +
  113. Uber Ringpop is an open- source Node.js library that brings

    application-layer sharding to many of their dispatching platform services. Swim Gossip Protocol Consistent Hashing +
  114. Orleans Cluster Orleans is a runtime and Programming model for

    building distributed systems based on the Actor Model from the eXtreme Computing Group at MSR Gossip Protocol Consistent Hashing + + Distributed Hash Table Actor Actor Actor Actor Actor
  115. Orleans Cluster Orleans is a runtime and Programming model for

    building distributed systems based on the Actor Model from the eXtreme Computing Group at MSR Gossip Protocol Consistent Hashing + + Distributed Hash Table Actor Actor Actor Actor Actor
  116. Orleans Cluster Orleans is a runtime and Programming model for

    building distributed systems based on the Actor Model from the eXtreme Computing Group at MSR Gossip Protocol Consistent Hashing + + Distributed Hash Table Actor Actor Actor Actor Actor
  117. Orleans Cluster Orleans Distributed Hash Table

  118. Orleans Cluster Orleans Distributed Hash Table

  119. Orleans Cluster Orleans Distributed Hash Table

  120. Orleans Cluster Orleans Distributed Hash Table

  121. Orleans Cluster Orleans Distributed Hash Table

  122. Orleans Cluster Orleans Distributed Hash Table

  123. Orleans Cluster Orleans Distributed Hash Table

  124. Orleans Cluster Orleans Distributed Hash Table

  125. Caution Stateful Services Ahead

  126. Unbounded Data Structures Implicit Assumptions are the Killer of Distributed

    Systems
  127. Memory Management Get Ready to Make Friends with the Garbage

    Collector Profiler
  128. Reloading State • First Connection • Recovering From Crashes •

    Deploying New Code
  129. Fast Restarts at Facebook “Our Key Observation is that we

    can decouple the memory lifetime from the process lifetime. When we shutdown a server for a planned upgrade.”
  130. Conclusion Data Locality & Available Consistency Cluster Membership & Work

    Distribution Successful Stateful Real World Systems Caution: Some New Challenges
  131. Thank You Kyle Kingsbury Chris Meiklejohn Tom Santero Ines Sombra

    @Caitie
  132. https://github.com/CaitieM20/ScalingStatefulServices @Caitie Questions

  133. None