Pro Yearly is on sale from $80 to $50! »

Distributed Sagas: A Protocol for Coordinating Microservices

Distributed Sagas: A Protocol for Coordinating Microservices

Microservices have become the defacto architecture pattern for building services. However separating business logic into small services that operate with a single logical data set has introduced consistency challenges. Previous attempts to solve this problem like two phase commit have not been widely adopted due to availability and liveness issues.

Instead developers implement feral concurrency control mechanism. This technique can be error prone, and often results in “Death Star” architectures which rely on chained calls to enforce application invariants. These architectures become more complicated over time, and are difficult to modify and extend, and often don't correctly handle all failure scenarios.

In this talk I propose a new solution for this problem, Distributed Sagas, a protocol for coordinating requests among multiple micro services, while ensuring application invariants.

9128d500301ae51524e887bb680f471d?s=128

Caitie McCaffrey

May 19, 2017
Tweet

Transcript

  1. Distributed Sagas A Protocol for Coordinating Microservices

  2. Caitie McCaffrey Distributed Systems Engineer caitiem.com @caitie

  3. Service Service Service Monoliths

  4. Microservices & NoSQL

  5. 2015

  6. 2015 “Application-level Mechanisms for maintaining database integrity”

  7. Reserve a Hotel Front End

  8. Reserve a Hotel Front End Feral Concurrency Control Mechanisms

  9. Reserve a Hotel Front End Feral Concurrency Control Mechanisms

  10. Reserve a Car Front End

  11. Reserve a Flight Front End

  12. Trips Front End

  13. Trips Front End Trips

  14. Trips Front End Trips Feral Concurrency Control Mechanisms

  15. Trips Front End Trips Feral Concurrency Control Mechanisms

  16. Trips Front End Trips Feral Concurrency Control Mechanisms

  17. Front End Trips Architecture

  18. Death Star Architectures

  19. Can We Do Better than Feral Concurrency Control?

  20. 2012

  21. 2012 “Spanner is Google’s scalable, multi- version, globally distributed, and

    synchronously-replicated database”
  22. 2015

  23. 2015 “The biggest barrier…is that consistency mechanisms must integrate across

    many stateful services”
  24. Two Phase Commit

  25. 2PC: Prepare Front End Trips Propose Book Hotel Propose Book

    Car Propose Book Flight
  26. 2PC: Prepare Front End Trips Vote Book Hotel Vote Book

    Car Vote Book Flight
  27. 2PC: Commit Front End Trips Commit/Abort Book Hotel Commit/Abort Book

    Car Commit/Abort Book Flight
  28. 2PC: Commit Front End Trips Done Done Done

  29. 2PC: Commit Front End Trips Done Done Done Doesn’t Scale

    •O (N^2) Messages in the worst case •Coordinator is a Single Point of Failure •Reduced Throughput
  30. Distributed Sagas A Protocol for Coordinating Microservices

  31. 1987

  32. 1987 “Sagas are Long Lived Transactions [in a single relational

    Database]”
  33. 1987 “A Saga is a Long Lived Transaction that can

    be written as a sequence of transactions that can be interleaved. All transactions in the sequence complete successfully or compensating transactions are ran to amend a partial execution.”
  34. Distributed Sagas A Protocol for Coordinating Microservices

  35. A Distributed Saga is a Collection of Requests Book Hotel

    Book Car Book Flight Charge Money
  36. A Distributed Saga is a Collection of Requests Book Hotel

    Book Car Book Flight Charge Money and Compensating Requests Cancel Hotel Cancel Car Cancel Flight Refund Money
  37. A Distributed Saga is a Collection of Requests Book Hotel

    Book Car Book Flight Charge Money and Compensating Requests Cancel Hotel Cancel Car Cancel Flight Refund Money that represent a single business level action
  38. Distributed Saga Requests

  39. Requests Can Abort Book Car

  40. Requests Can Abort Book Car

  41. Requests Must Be Idempotent Book Car

  42. Requests Must Be Idempotent Book Car

  43. Requests Must Be Idempotent Book Car

  44. Requests Must Be Idempotent Book Car

  45. Requests Must Be Idempotent Book Car

  46. Requests Must Be Idempotent Book Car Timeout

  47. Requests Must Be Idempotent Book Car

  48. Requests Must Be Idempotent Book Car

  49. Distributed Saga Compensating Requests

  50. Compensating Requests Semantically undoes the effect of a request

  51. Cancel Car Compensating Requests Cannot Abort

  52. Cancel Car Compensating Requests Can Not Abort

  53. Cancel Car Compensating Requests Must Be Idempotent

  54. Compensating Requests Must Be Commutative with Requests Book Car Cancel

    Car Cancel Car Book Car is the same as
  55. Compensating Requests Book Car Must Be Commutative with Requests

  56. Compensating Requests Book Car Timeout Must Be Commutative with Requests

  57. Compensating Requests Book Car Book Car Must Be Commutative with

    Requests
  58. Compensating Requests Book Car Book Car Cancel Car Must Be

    Commutative with Requests
  59. Compensating Requests Book Car Book Car Cancel Car Must Be

    Commutative with Requests
  60. Compensating Requests Requests Idempotent Idempotent Commutative Can Not Abort Can

    Abort
  61. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge

    Money All requests were completed successfully
  62. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge

    Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car
  63. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge

    Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car No Atomicity No Isolation
  64. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge

    Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car No Atomicity No Isolation Book Hotel Book Car Visible before Saga Completes
  65. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge

    Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car
  66. Distributed Saga defining a

  67. Directed Acyclic Graph Distributed Saga Car Flight Hotel Payment Start

    Saga End Saga
  68. Name: Hotel Request: Book Hotel Compensating Request: Cancel Hotel Status:

    Not Completed Vertex Distributed Saga
  69. Start & End Vertices Distributed Saga Start Saga End Saga

    Car Flight Hotel Payment Start Saga End Saga
  70. Distributed Saga Log fault-tolerant & highly available

  71. Saga Execution Coordinator SEC Car Flight Hotel Payment Start Saga

    End Saga Saga Log
  72. Distributed Saga Executing a

  73. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Book Trip Request SEC
  74. Car Flight Hotel Payment Start Saga End Saga SEC Saga

    Log Start Saga Book Trip Request
  75. Car Flight Hotel Payment Start Saga End Saga SEC Saga

    Log Done Start Saga Book Trip Request
  76. Car Flight Hotel Payment Start Saga End Saga Saga Log

    SEC Start Hotel Start Saga
  77. Car Flight Hotel Payment Start Saga End Saga Saga Log

    SEC Done Start Saga Start Hotel
  78. Car Flight Hotel Payment Start Saga End Saga Saga Log

    SEC Book Hotel Request Start Saga Start Hotel
  79. Car Flight Hotel Payment Start Saga End Saga Saga Log

    SEC Book Hotel Response Start Saga Start Hotel
  80. Car Flight Hotel Payment Start Saga End Saga Saga Log

    SEC End Hotel Start Saga Start Hotel
  81. Car Flight Hotel Payment Start Saga End Saga Saga Log

    SEC Done Start Saga Start Hotel End Hotel
  82. Car Flight Hotel Payment Start Saga End Saga Saga Log

    SEC Start Car Start Saga Start Hotel End Hotel
  83. Car Flight Hotel Payment Start Saga End Saga Saga Log

    SEC Done Book Car Request Start Saga Start Hotel End Hotel Start Car
  84. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car Book Car Response End Car
  85. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car Done End Car
  86. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car Start Flight End Car
  87. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car Done End Car Start Flight Book Flight Request
  88. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight Book Flight Response End Flight
  89. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight Done End Flight
  90. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight Start Payment End Flight
  91. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight Done End Flight Start Payment Payment Request
  92. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Payment End Flight Start Payment Payment Response
  93. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Flight Start Payment End Payment Done
  94. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Flight Start Payment End Payment End Saga
  95. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Flight Start Payment End Payment Done End Saga
  96. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Flight Start Payment End Payment End Saga
  97. Distributed Saga Failure of a

  98. Car Flight Hotel Payment Start Saga End Saga Saga Log

    SEC Book Car Request Start Saga Start Hotel End Hotel Start Car Start Flight
  99. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Book Car Response Abort Car Start Car Start Flight
  100. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Done Start Car Start Flight Abort Car
  101. Car Flight Hotel Payment Start Saga End Saga Saga Log

    Start Saga SEC Start Hotel End Hotel Done Start Car Start Flight Abort Car Rollback Recovery
  102. Car Flight Hotel Payment End Comp Saga Start Comp Saga

    Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car
  103. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Start Comp Saga End Comp Saga
  104. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Payment Log Entries? Start Car Start Flight Abort Car Start Comp Saga End Comp Saga
  105. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel { } Start Car Start Flight Abort Car Start Comp Saga End Comp Saga
  106. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Car Log Entries? Start Comp Saga End Comp Saga
  107. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car { Start, Abort } Start Comp Saga End Comp Saga
  108. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Hotel Log Entries? Start Comp Saga End Comp Saga
  109. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car { Start, End } Start Comp Saga End Comp Saga
  110. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car { Start, End } Cancel Hotel Request Start Comp Saga End Comp Saga
  111. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Cancel Hotel Response Comp Hotel Start Comp Saga End Comp Saga
  112. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Done Comp Hotel Start Comp Saga End Comp Saga
  113. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Flight Log Entries? Start Comp Saga End Comp Saga
  114. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel { Start } Start Comp Saga End Comp Saga
  115. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel { Start } Book Flight Request Start Comp Saga End Comp Saga
  116. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel End Flight Book Flight Response Start Comp Saga End Comp Saga
  117. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Done Cancel Flight Request End Flight Start Comp Saga End Comp Saga
  118. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Comp Flight Cancel Flight Response Start Comp Saga End Comp Saga End Flight
  119. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Done Comp Flight Start Comp Saga End Comp Saga End Flight Comp Flight
  120. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel End Saga Start Comp Saga End Comp Saga Comp Flight End Flight Comp Flight
  121. Car Flight Hotel Payment Saga Log Start Saga SEC Start

    Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Done End Saga Start Comp Saga End Comp Saga Comp Flight End Flight Comp Flight
  122. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge

    Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car
  123. Saga Execution Coordinator Recovering from Failure

  124. Front End SEC Start Saga End Saga Saga Log

  125. Front End SEC Start Saga End Saga Saga Log

  126. Front End SEC Start Saga End Saga Saga Log

  127. Front End SEC Saga Log Start Saga End Saga

  128. with Distributed Sagas Isolation of Complex Code Fron Tr Complex

    Code Lives Everywhere Front SEC Saga Log Complex Code Lives Here Complex Code Lives Here
  129. Fron Tr with Distributed Sagas Modular Services Front SEC Saga

    Log Only Flight Booking Flight Booking & Feral Concurrency Control Mechanisms
  130. Fron Tr with Distributed Sagas Service Composition Front SEC Saga

    Log Just Add A New Saga! New Service & Feral Concurrency Control Mechanisms
  131. Distributed Sagas Makes Building & Modifying Microservices Easier

  132. Thank you @caitie