Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Scalable Stateful Services

Building Scalable Stateful Services

The Stateless Service design principle has become ubiquitous in the tech industry for creating horizontally scalable services. However our applications do have state, we just have moved all of it to caches and databases. Today as applications are becoming more data intensive and request latencies are expected to be incredibly low, we’d like the benefits of stateful services, like data locality and sticky consistency. In this talk I will address the benefits of stateful services, how to build them so that they scale, and discuss projects from Halo and Twitter of highly distributed and scalable services that implement these techniques successfully.

Caitie McCaffrey

September 27, 2015
Tweet

More Decks by Caitie McCaffrey

Other Decks in Programming

Transcript

  1. Building Scalable Stateful Services
    StrangeLoop 2015

    View full-size slide

  2. Caitie McCaffrey
    Distributed Systems Engineer
    Tech Lead Observability @ Twitter
    @caitie
    caitiem.com

    View full-size slide

  3. Stateless Services
    Service

    View full-size slide

  4. Stateless Services
    Service

    View full-size slide

  5. Stateless Services
    Service
    Service

    View full-size slide

  6. Stateless Services
    Service
    Service

    View full-size slide

  7. Stateless Services
    Service
    Service

    View full-size slide

  8. Stateless Services
    Service Service
    Service

    View full-size slide

  9. Stateless Services
    Service Service
    Service

    View full-size slide

  10. Stateless Services
    Service Service
    Service

    View full-size slide

  11. Stateless Services
    Service Service
    Service
    Service

    View full-size slide

  12. Stateless Services
    Service Service
    Service
    Service

    View full-size slide

  13. Stateless Services
    Service Service
    Service
    Service

    View full-size slide

  14. Stateless Services
    Service Service
    Service
    Service Service

    View full-size slide

  15. Stateless Services
    Service Service
    Service
    Service Service

    View full-size slide

  16. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  17. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  18. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  19. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  20. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  21. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  22. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  23. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  24. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  25. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  26. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  27. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  28. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  29. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  30. Data Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  31. Overview
    Benefits
    Building
    Real World
    Caution

    View full-size slide

  32. Data Locality
    For Low Latency & Data Intensive Services

    View full-size slide

  33. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  34. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  35. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  36. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  37. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  38. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  39. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  40. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  41. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  42. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  43. Function Shipping Paradigm
    Service
    Service
    Service

    View full-size slide

  44. Sticky Connections
    & Consistency
    Additional Available Consistency Models

    View full-size slide

  45. Linearizable
    Sequential
    Causal
    Pipelined Random
    Access Memory
    Read Your
    Write
    Monotonic
    Read
    Monotonic
    Write
    Write From
    Read
    Consistency
    Models
    CP Consistency
    AP Consistency

    View full-size slide

  46. Linearizable
    Sequential
    Causal
    Pipelined Random
    Access Memory
    Read Your
    Write
    Monotonic
    Read
    Monotonic
    Write
    Write From
    Read
    CP Consistency
    AP Consistency
    AP Consistency
    w/ Sticky Connections
    Consistency
    Models

    View full-size slide

  47. - Werner Vogel 2007
    “Whether or not read-your-write, session and monotonic
    consistency can be achieved depends in general on the
    "stickiness" of clients to the server that executes the
    distributed protocol for them… Using sessions, which are
    sticky, makes this explicit and provides an exposure level
    that clients can reason about.”

    View full-size slide

  48. Building Sticky Connections
    For Low Latency & Data Intensive Services

    View full-size slide

  49. Building Sticky Connections

    View full-size slide

  50. Building Sticky Connections

    View full-size slide

  51. Persistent Connections
    Load Balancing Problems
    No Stickiness Once
    Connection Breaks
    Problems

    View full-size slide

  52. Persistent Connections
    Load Balancing Problems
    No Stickiness Once
    Connection Breaks
    Problems

    View full-size slide

  53. Routing Logic
    • Cluster Membership
    • Work Distribution
    Problems to Solve

    View full-size slide

  54. Routing Logic
    • Cluster Membership
    • Work Distribution
    Problems to Solve

    View full-size slide

  55. Routing Logic
    • Cluster Membership
    • Work Distribution
    Problems to Solve

    View full-size slide

  56. Static Cluster Membership

    View full-size slide

  57. Static Cluster Membership

    View full-size slide

  58. Static Cluster Membership

    View full-size slide

  59. Dynamic Cluster Membership

    View full-size slide

  60. Dynamic Cluster Membership

    View full-size slide

  61. Dynamic Cluster Membership

    View full-size slide

  62. Dynamic Cluster Membership

    View full-size slide

  63. Dynamic Cluster Membership

    View full-size slide

  64. Dynamic Cluster Membership
    Gossip
    Protocols
    Consensus
    Systems
    Availability vs Consistency

    View full-size slide

  65. Work Distribution
    Consistent
    Hashing
    Distributed
    Hash Tables
    Random
    Placement

    View full-size slide

  66. Random Placement
    Write
    Anywhere
    Read from
    Everywhere

    View full-size slide

  67. Random Placement
    Write
    Anywhere
    Read from
    Everywhere

    View full-size slide

  68. Random Placement
    Write
    Anywhere
    Read from
    Everywhere

    View full-size slide

  69. Random Placement
    Write
    Anywhere
    Read from
    Everywhere

    View full-size slide

  70. Random Placement
    Write
    Anywhere
    Read from
    Everywhere

    View full-size slide

  71. Random Placement
    Write
    Anywhere
    Read from
    Everywhere

    View full-size slide

  72. Random Placement
    Write
    Anywhere
    Read from
    Everywhere

    View full-size slide

  73. Consistent Hashing
    Deterministic
    Placement
    Node A
    Node B
    Node C
    NodeD
    Consistent Hashing & Random Trees:
    Distributed caching protocols for
    relieving hot spots on the World
    Wide Web

    View full-size slide

  74. Distributed Hash Table
    Non- Deterministic
    Placement
    Node B
    Node A
    Node C

    View full-size slide

  75. Distributed Hash Table
    Non- Deterministic
    Placement
    Node B
    Node A
    Node C

    View full-size slide

  76. Distributed Hash Table
    Non- Deterministic
    Placement
    Node B
    Node A
    Node C

    View full-size slide

  77. Distributed Hash Table
    Non- Deterministic
    Placement
    Node B
    Node A
    Node C

    View full-size slide

  78. Distributed Hash Table
    Non- Deterministic
    Placement
    Node B
    Node A
    Node C

    View full-size slide

  79. Stateful Services
    In the Real World

    View full-size slide

  80. Scuba is a fast, scalable,
    distributed, in-memory database
    built at Facebook. It is the workhorse
    behind code regression analysis &
    bug report, revenue, and
    performance debugging
    Fan-out request to all
    machines in the cluster
    Compose Results
    Return Results and
    Completeness

    View full-size slide

  81. Scuba is a fast, scalable,
    distributed, in-memory database
    built at Facebook. It is the workhorse
    behind code regression analysis &
    bug report, revenue, and
    performance debugging
    Fan-out request to all
    machines in the cluster
    Compose Results
    Return Results and
    Completeness

    View full-size slide

  82. Scuba is a fast, scalable,
    distributed, in-memory database
    built at Facebook. It is the workhorse
    behind code regression analysis &
    bug report, revenue, and
    performance debugging
    Fan-out request to all
    machines in the cluster
    Compose Results
    Return Results and
    Completeness

    View full-size slide

  83. Uber Ringpop is an open-
    source Node.js library that brings
    application-layer sharding to
    many of their dispatching
    platform services.
    Swim Gossip Protocol
    Consistent Hashing
    +

    View full-size slide

  84. Uber Ringpop is an open-
    source Node.js library that brings
    application-layer sharding to
    many of their dispatching
    platform services.
    Swim Gossip Protocol
    Consistent Hashing
    +

    View full-size slide

  85. Uber Ringpop is an open-
    source Node.js library that brings
    application-layer sharding to
    many of their dispatching
    platform services.
    Swim Gossip Protocol
    Consistent Hashing
    +

    View full-size slide

  86. Orleans Cluster
    Orleans is a runtime and
    Programming model for building
    distributed systems based on the
    Actor Model from the eXtreme
    Computing Group at MSR
    Gossip Protocol
    Consistent Hashing
    +
    +
    Distributed Hash Table
    Actor
    Actor
    Actor
    Actor
    Actor

    View full-size slide

  87. Orleans Cluster
    Orleans is a runtime and
    Programming model for building
    distributed systems based on the
    Actor Model from the eXtreme
    Computing Group at MSR
    Gossip Protocol
    Consistent Hashing
    +
    +
    Distributed Hash Table
    Actor
    Actor
    Actor
    Actor
    Actor

    View full-size slide

  88. Orleans Cluster
    Orleans is a runtime and
    Programming model for building
    distributed systems based on the
    Actor Model from the eXtreme
    Computing Group at MSR
    Gossip Protocol
    Consistent Hashing
    +
    +
    Distributed Hash Table
    Actor
    Actor
    Actor
    Actor
    Actor

    View full-size slide

  89. Orleans Cluster
    Orleans Distributed Hash Table

    View full-size slide

  90. Orleans Cluster
    Orleans Distributed Hash Table

    View full-size slide

  91. Orleans Cluster
    Orleans Distributed Hash Table

    View full-size slide

  92. Orleans Cluster
    Orleans Distributed Hash Table

    View full-size slide

  93. Orleans Cluster
    Orleans Distributed Hash Table

    View full-size slide

  94. Orleans Cluster
    Orleans Distributed Hash Table

    View full-size slide

  95. Orleans Cluster
    Orleans Distributed Hash Table

    View full-size slide

  96. Orleans Cluster
    Orleans Distributed Hash Table

    View full-size slide

  97. Caution
    Stateful Services Ahead

    View full-size slide

  98. Unbounded
    Data Structures
    Implicit Assumptions
    are the Killer of
    Distributed Systems

    View full-size slide

  99. Memory Management
    Get Ready to Make Friends with the Garbage Collector Profiler

    View full-size slide

  100. Reloading
    State
    • First Connection
    • Recovering From Crashes
    • Deploying New Code

    View full-size slide

  101. Fast Restarts at Facebook
    “Our Key Observation is that we can
    decouple the memory lifetime from the
    process lifetime. When we shutdown a
    server for a planned upgrade.”

    View full-size slide

  102. Conclusion
    Data Locality & Available Consistency
    Cluster Membership & Work Distribution
    Successful Statefull Real World Systems
    Caution: Some New Challenges

    View full-size slide

  103. Should I Read Papers?

    View full-size slide

  104. Should I Read Papers?
    YES!

    View full-size slide

  105. Thank You
    Kyle Kingsbury
    Chris Meiklejohn
    Tom Santero
    Ines Sombra
    @Caitie

    View full-size slide