CraftConf 2016 - Building Scalable Stateful Services

The Stateless Service design principle has become ubiquitous in the tech industry for creating horizontally scalable services. However our applications do have state, we just have moved all of it to caches and databases. Today as applications are becoming more data intensive and request latencies are expected to be incredibly low, we’d like the benefits of stateful services, like data locality and sticky consistency. In this talk I will address the benefits of stateful services, how to build them so that they scale, and discuss distributed and scalable services in the real world that implement these techniques successfully.

Transcript

1. Building Scalable Stateful Services Craft Conf 2016
2. Caitie McCaffrey Distributed Systems Engineer @caitie caitiem.com
3. Stateless Services Service
4. Stateless Services Service
5. Stateless Services Service Service
6. Stateless Services Service Service
7. Stateless Services Service Service
8. Stateless Services Service Service Service
9. Stateless Services Service Service Service
10. Stateless Services Service Service Service
11. Stateless Services Service Service Service Service
12. Stateless Services Service Service Service Service
13. Stateless Services Service Service Service Service
14. Stateless Services Service Service Service Service Service
15. Stateless Services Service Service Service Service Service
16. Data Shipping Paradigm Service Service Service
17. Data Shipping Paradigm Service Service Service
18. Data Shipping Paradigm Service Service Service
19. Data Shipping Paradigm Service Service Service
20. Data Shipping Paradigm Service Service Service
21. Data Shipping Paradigm Service Service Service
22. Data Shipping Paradigm Service Service Service
23. Data Shipping Paradigm Service Service Service
24. Data Shipping Paradigm Service Service Service
25. Data Shipping Paradigm Service Service Service
26. Data Shipping Paradigm Service Service Service
27. Data Shipping Paradigm Service Service Service
28. Data Shipping Paradigm Service Service Service
29. Data Shipping Paradigm Service Service Service
30. Data Shipping Paradigm Service Service Service
31. Overview Benefits Building Real World Caution
32. Data Locality For Low Latency & Data Intensive Services
33. Function Shipping Paradigm Service Service Service
34. Function Shipping Paradigm Service Service Service
35. Function Shipping Paradigm Service Service Service
36. Function Shipping Paradigm Service Service Service
37. Function Shipping Paradigm Service Service Service
38. Function Shipping Paradigm Service Service Service
39. Function Shipping Paradigm Service Service Service
40. Function Shipping Paradigm Service Service Service
41. Function Shipping Paradigm Service Service Service
42. Function Shipping Paradigm Service Service Service
43. Function Shipping Paradigm Service Service Service
44. Sticky Connections & Consistency Additional Available Consistency Models
45. Sticky Connections
46. Sticky Connections
47. Linearizable Sequential Causal Pipelined Random Access Memory Read Your Write Monotonic Read Monotonic Write Write From Read Consistency Models CP Consistency AP Consistency
48. Linearizable Sequential Causal Pipelined Random Access Memory Read Your Write Monotonic Read Monotonic Write Write From Read CP Consistency AP Consistency AP Consistency w/ Sticky Connections Consistency Models
49. - Werner Vogel 2007 “Whether or not read-your-write, session and monotonic consistency can be achieved depends in general on the "stickiness" of clients to the server that executes the distributed protocol for them… Using sessions, which are sticky, makes this explicit and provides an exposure level that clients can reason about.”
50. Building Sticky Connections For Low Latency & Data Intensive Services
51. Building Sticky Connections
52. Building Sticky Connections
53. Persistent Connections Load Balancing Problems No Stickiness Once Connection Breaks Problems
54. Persistent Connections Load Balancing Problems No Stickiness Once Connection Breaks Problems
55. Routing Logic • Cluster Membership • Work Distribution Problems to Solve
56. Routing Logic • Cluster Membership • Work Distribution Problems to Solve
57. Routing Logic • Cluster Membership • Work Distribution Problems to Solve
58. Cluster Membership Dynamic Static
59. Static Cluster Membership Machine Outage means Partial Service Outage Downtime to Add Capacity
60. Static Cluster Membership Machine Outage means Partial Service Outage Downtime to Add Capacity
61. Static Cluster Membership Machine Outage means Partial Service Outage Downtime to Add Capacity
62. Dynamic Cluster Membership Fault Tolerance
63. Dynamic Cluster Membership Fault Tolerance
64. Dynamic Cluster Membership Fault Tolerance
65. Dynamic Cluster Membership Fault Tolerance
66. Dynamic Cluster Membership Fault Tolerance
67. Dynamic Cluster Membership Gossip Protocols Consensus Systems Availability vs Consistency
68. Consensus Systems for Consistency Paxos ETCD Zookeeper Raft
69. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
70. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
71. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
72. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
73. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
74. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
75. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
76. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
77. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
78. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
79. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
80. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
81. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
82. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
83. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
84. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
85. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
86. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
87. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
88. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
89. Gossip Protocols for Availability • Assumes Non-Reliable Networks • Information Dissemination • Pairwise Communication
90. Work Distribution Consistent Hashing Distributed Hash Tables Random Placement
91. Random Placement Write Anywhere Read from Everywhere
92. Random Placement Write Anywhere Read from Everywhere
93. Random Placement Write Anywhere Read from Everywhere
94. Random Placement Write Anywhere Read from Everywhere
95. Random Placement Write Anywhere Read from Everywhere
96. Random Placement Write Anywhere Read from Everywhere
97. Random Placement Write Anywhere Read from Everywhere
98. Consistent Hashing Deterministic Placement Node A Node B Node C NodeD Consistent Hashing & Random Trees: Distributed caching protocols for relieving hot spots on the World Wide Web
99. Distributed Hash Table Non- Deterministic Placement Node B Node A Node C
100. Distributed Hash Table Non- Deterministic Placement Node B Node A Node C
101. Distributed Hash Table Non- Deterministic Placement Node B Node A Node C
102. Distributed Hash Table Non- Deterministic Placement Node B Node A Node C
103. Distributed Hash Table Non- Deterministic Placement Node B Node A Node C
104. Stateful Services In the Real World
105. Scuba is a fast, scalable, distributed, in-memory database built at Facebook. It is the workhorse behind code regression analysis & bug report, revenue, and performance debugging Fan-out request to all machines in the cluster Compose Results Return Results and Completeness
106. Scuba is a fast, scalable, distributed, in-memory database built at Facebook. It is the workhorse behind code regression analysis & bug report, revenue, and performance debugging Fan-out request to all machines in the cluster Compose Results Return Results and Completeness
107. Scuba is a fast, scalable, distributed, in-memory database built at Facebook. It is the workhorse behind code regression analysis & bug report, revenue, and performance debugging Fan-out request to all machines in the cluster Compose Results Return Results and Completeness
108. Twitter Nuthatch is a temporal set indexing service. Helps resolve glob queries in metric names. Static Cluster Membership + Hashing
109. Twitter Nuthatch is a temporal set indexing service. Helps resolve glob queries in metric names. Static Cluster Membership + Hashing
110. Twitter Nuthatch is a temporal set indexing service. Helps resolve glob queries in metric names. Static Cluster Membership + Hashing
111. Uber Ringpop is an open- source Node.js library that brings application-layer sharding to many of their dispatching platform services. Swim Gossip Protocol Consistent Hashing +
112. Uber Ringpop is an open- source Node.js library that brings application-layer sharding to many of their dispatching platform services. Swim Gossip Protocol Consistent Hashing +
113. Uber Ringpop is an open- source Node.js library that brings application-layer sharding to many of their dispatching platform services. Swim Gossip Protocol Consistent Hashing +
114. Orleans Cluster Orleans is a runtime and Programming model for building distributed systems based on the Actor Model from the eXtreme Computing Group at MSR Gossip Protocol Consistent Hashing + + Distributed Hash Table Actor Actor Actor Actor Actor
115. Orleans Cluster Orleans is a runtime and Programming model for building distributed systems based on the Actor Model from the eXtreme Computing Group at MSR Gossip Protocol Consistent Hashing + + Distributed Hash Table Actor Actor Actor Actor Actor
116. Orleans Cluster Orleans is a runtime and Programming model for building distributed systems based on the Actor Model from the eXtreme Computing Group at MSR Gossip Protocol Consistent Hashing + + Distributed Hash Table Actor Actor Actor Actor Actor
117. Orleans Cluster Orleans Distributed Hash Table
118. Orleans Cluster Orleans Distributed Hash Table
119. Orleans Cluster Orleans Distributed Hash Table
120. Orleans Cluster Orleans Distributed Hash Table
121. Orleans Cluster Orleans Distributed Hash Table
122. Orleans Cluster Orleans Distributed Hash Table
123. Orleans Cluster Orleans Distributed Hash Table
124. Orleans Cluster Orleans Distributed Hash Table
125. Caution Stateful Services Ahead
126. Unbounded Data Structures Implicit Assumptions are the Killer of Distributed Systems
127. Memory Management Get Ready to Make Friends with the Garbage Collector Profiler
128. Reloading State • First Connection • Recovering From Crashes • Deploying New Code
129. Fast Restarts at Facebook “Our Key Observation is that we can decouple the memory lifetime from the process lifetime. When we shutdown a server for a planned upgrade.”
130. Conclusion Data Locality & Available Consistency Cluster Membership & Work Distribution Successful Stateful Real World Systems Caution: Some New Challenges
131. Thank You Kyle Kingsbury Chris Meiklejohn Tom Santero Ines Sombra @Caitie
132. https://github.com/CaitieM20/ScalingStatefulServices @Caitie Questions