Applying the Saga Pattern

As we build larger more complex applications and solutions that need to do collaborative processing the traditional ACID transaction model using coordinated 2-phase commit is often no longer suitable. More frequently we have long lived transactions or must act upon resources distributed across various locations and trust boundaries. The Saga Pattern is a useful model for long lived activities and distributed transactions without coordination.

Sagas split work into a set of transactions whose effects can be reversed even after the work has been performed or committed. If a failure occurs compensating transactions are performed to rollback the work. So at its core the Saga is a failure Management Pattern, making it particularly applicable to distributed systems.

In this talk, I'll discuss the fundamentals of the Saga Pattern, and how it can be applied to your systems. In addition we'll discuss how the Halo 4 Services successfully made use of the Saga Pattern when processing game statistics, and how we implemented it in production.

Transcript

1. Applying the Saga Pattern
   Caitie McCafffrey
   

   View Slide

2. Caitie McCaffrey!
   Distributed Systems Engineer
   @Caitie
   CaitieM.com
   

   View Slide

3. View Slide

4. • Why Sagas?
   • Sagas Paper
   • Distributed Sagas
   • Sagas in Halo 4
   

   View Slide

5. Systems Used to Be Simple
   

   View Slide

6. Serializability &
   ACID
   

   View Slide

7. SOA/Microservices
   

   View Slide

8. Two Phase Commit
   

   View Slide

9. 2PC: Prepare Phase
   

   View Slide

10. 2PC: Commit Phase
    

    View Slide

11. 2PC Doesn’t Scale
    • O(n^2) Messages
    • Coordinator: Single Point of Failure
    • Reduced Throughput
    

    View Slide

12. Spanner
    Google’s Globally-Distributed Database
    Corbett et. al.
    

    View Slide

13. –Corbett et al.
    “Spanner is Google’s scalable, multi-version,
    globally distributed, and synchronously-
    replicated database. It is the first system to
    distribute data at global scale and support
    externally-consistent distributed
    transactions.”
    

    View Slide

14. “The key enabler of these
    properties is a new TrueTime API
    and its implementation…using
    multiple modern clock references
    (GPS and atomic clocks).”
    –Corbett et al.
    

    View Slide

15. Spanner is Expensive & Proprietary
    • Custom Hardware & Data-Centers
    • Synchronization Not Solved
    

    View Slide

16. Distributed Transactions
    are Hard & Expensive
    

    View Slide

17. Can We Do
    Better?
    

    View Slide

18. Can We Do
    Better?
    

    View Slide

19. Sagas
    Hector Garcia-Molina, Kenneth Salem
    Princeton University 1987
    

    View Slide

20. Sagas are Long
    Lived Transactions
    

    View Slide

21. “A Saga is a Long Lived Transaction that
    can be written as a sequence of
    transactions that can be interleaved.
    All transactions in the sequence complete
    successfully or compensating transactions
    are ran to amend a partial execution.”
    

    View Slide

22. A Saga is a Collection of
    Sub-Transactions
    T1, T2 … Tn
    

    View Slide

23. Each Sub-Transaction has a
    Compensating Transaction
    C1, C2 … Cn
    

    View Slide

24. Cn
    Semantically Undoes Tn
    

    View Slide

25. Saga Guarantee
    Either
    • T1, T2 … Tn or
    • T1, T2 … Tj, Cj, … C2, C1
    

    View Slide

26. Trade-Off: Atomicity
    for Availability
    

    View Slide

27. Sagas are a Failure
    Management Pattern
    

    View Slide

28. Large Single Transaction
    

    View Slide

29. • Book Hotel (T1
    )
    • Book Car (T2
    )
    • Book Flight (T3
    )
    • Cancel Hotel (C1
    )
    • Cancel Car (C2
    )
    • Cancel Flight (C3
    )
    Sagas
    

    View Slide

30. Saga Execution
    Coordinator
    (SEC)
    

    View Slide

31. Saga Log
    • Begin Saga
    • End Saga
    • Abort Saga
    • Begin Ti
    • End Ti
    • Begin Ci
    • End Ci
    

    View Slide

32. Begin Saga!
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    End Book Car Rental (T2
    )
    Start Book Flight (T3
    )
    End Book Flight (T3
    )
    End Saga
    Successful Saga
    

    View Slide

33. Begin Saga
    Start Book Hotel (T1
    )!
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    End Book Car Rental (T2
    )
    Start Book Flight (T3
    )
    End Book Flight (T3
    )
    End Saga
    Successful Saga
    

    View Slide

34. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )!
    Start Book Car Rental (T2
    )
    End Book Car Rental (T2
    )
    Start Book Flight (T3
    )
    End Book Flight (T3
    )
    End Saga
    Successful Saga
    

    View Slide

35. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )!
    End Book Car Rental (T2
    )
    Start Book Flight (T3
    )
    End Book Flight (T3
    )
    End Saga
    Successful Saga
    

    View Slide

36. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    End Book Car Rental (T2
    )!
    Start Book Flight (T3
    )
    End Book Flight (T3
    )
    End Saga
    Successful Saga
    

    View Slide

37. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    End Book Car Rental (T2
    )
    Start Book Flight (T3
    )!
    End Book Flight (T3
    )
    End Saga
    Successful Saga
    

    View Slide

38. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    End Book Car Rental (T2
    )
    Start Book Flight (T3
    )
    End Book Flight (T3
    )!
    End Saga
    Successful Saga
    

    View Slide

39. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    End Book Car Rental (T2
    )
    Start Book Flight (T3
    )
    End Book Flight (T3
    )
    End Saga
    Successful Saga
    

    View Slide

40. Unsuccessful Saga
    Backwards Recovery
    

    View Slide

41. Begin Saga!
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    Abort Saga
    Start Compensate Car Rental (C2
    )
    End Compensate Car Rental (C2
    )
    Start Compensate Book Hotel (C1
    )
    End Compensate Book Hotel (C1
    )
    End Saga
    Unsuccessful Saga
    

    View Slide

42. Begin Saga
    Start Book Hotel (T1
    )!
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    Abort Saga
    Start Compensate Car Rental (C2
    )
    End Compensate Car Rental (C2
    )
    Start Compensate Book Hotel (C1
    )
    End Compensate Book Hotel (C1
    )
    End Saga
    Unsuccessful Saga
    

    View Slide

43. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )!
    Start Book Car Rental (T2
    )
    Abort Saga
    Start Compensate Car Rental (C2
    )
    End Compensate Car Rental (C2
    )
    Start Compensate Book Hotel (C1
    )
    End Compensate Book Hotel (C1
    )
    End Saga
    Unsuccessful Saga
    

    View Slide

44. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )!
    Abort Saga
    Start Compensate Car Rental (C2
    )
    End Compensate Car Rental (C2
    )
    Start Compensate Book Hotel (C1
    )
    End Compensate Book Hotel (C1
    )
    End Saga
    Unsuccessful Saga
    

    View Slide

45. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    Abort Saga!
    Start Compensate Car Rental (C2
    )
    End Compensate Car Rental (C2
    )
    Start Compensate Book Hotel (C1
    )
    End Compensate Book Hotel (C1
    )
    End Saga
    Unsuccessful Saga
    

    View Slide

46. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    Abort Saga
    Start Compensate Car Rental (C2
    )!
    End Compensate Car Rental (C2
    )
    Start Compensate Book Hotel (C1
    )
    End Compensate Book Hotel (C1
    )
    End Saga
    Unsuccessful Saga
    

    View Slide

47. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    Abort Saga
    Start Compensate Car Rental (C2
    )
    End Compensate Car Rental (C2
    )!
    Start Compensate Book Hotel (C1
    )
    End Compensate Book Hotel (C1
    )
    End Saga
    Unsuccessful Saga
    

    View Slide

48. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    Abort Saga
    Start Compensate Car Rental (C2
    )
    End Compensate Car Rental (C2
    )
    Start Compensate Book Hotel (C1
    )!
    End Compensate Book Hotel (C1
    )
    End Saga
    Unsuccessful Saga
    

    View Slide

49. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    Abort Saga
    Start Compensate Car Rental (C2
    )
    End Compensate Car Rental (C2
    )
    Start Compensate Book Hotel (C1
    )
    End Compensate Book Hotel (C1
    )!
    End Saga
    Unsuccessful Saga
    

    View Slide

50. Begin Saga
    Start Book Hotel (T1
    )
    End Book Hotel (T1
    )
    Start Book Car Rental (T2
    )
    Abort Saga
    Start Compensate Car Rental (C2
    )
    End Compensate Car Rental (C2
    )
    Start Compensate Book Hotel (C1
    )
    End Compensate Book Hotel (C1
    )
    End Saga
    Unsuccessful Saga
    

    View Slide

51. –Molina et. al
    “Due to space limitations, we only
    discuss Sagas in a centralized
    System, although clearly they can
    be implemented in a distributed
    database system.”
    Sagas in Distributed Systems
    

    View Slide

52. SOA/Microservices
    

    View Slide

53. \
    • Book Hotel (T1
    )
    • Book Car (T2
    )
    • Book Flight (T3
    )
    • Cancel Hotel (C1
    )
    • Cancel Car (C2
    )
    • Cancel Flight (C3
    )
    Requests instead of Transactions
    

    View Slide

54. A Distributed Saga is a Collection of
    Sub-Requests
    Each Sub-Request has a
    Compensating Request
    T1, T2 … Tn
    C1, C2 … Cn
    

    View Slide

55. Begin Saga
    Start Book Hotel Request (T1
    )
    End Book Hotel Request (T1
    )
    Start Book Car Rental Request (T2
    )
    End Book Car Rental Request (T2
    )
    Start Book Flight Request (T3
    )
    End Book Flight Request (T3
    )
    End Saga
    Successful Distributed Saga
    

    View Slide

56. Saga Log
    Durable & Distributed
    

    View Slide

57. Saga Execution Coordinator (SEC)
    • Interprets & Writes to Saga Log
    • Applies Saga Sub-Requests
    • Applies Saga Compensating Requests
    when Necessary
    

    View Slide

58. View Slide

59. View Slide

60. View Slide

61. View Slide

62. View Slide

63. View Slide

64. View Slide

65. View Slide

66. View Slide

67. Apply Compensating Requests
    • Aborted Saga Response
    • Start Request Fails
    • SEC Crashes (non-safe state)
    

    View Slide

68. View Slide

69. View Slide

70. View Slide

71. What Happens when
    Compensating
    Requests Fail?
    

    View Slide

72. Compensating
    Requests Must Be
    Idempotent
    

    View Slide

73. What Happens
    when SEC Fails?
    

    View Slide

74. Safe States
    • All Executed Sub-Requests are Complete
    (Start Ti
    & End Ti
    both logged)
    • Saga has been Aborted, Proceed with
    Compensating Transactions
    

    View Slide

75. Un-Safe State
    • Start Ti logged, no End Ti logged
    Abort Saga
    Start Compensating Requests
    

    View Slide

76. Request Messaging Semantics
    • Sub-Requests (Ti): At Most Once
    • Compensating Requests (Ci): At Least Once
    

    View Slide

77. Distributed Saga Guarantee
    Either
    • T1, T2 … Tn or
    • T1, T2 … Tj, Cj, … C2, C1
    

    View Slide

78. Distributed Sagas
    • Distributed/Durable Saga Log
    • SEC Process
    • Compensating Requests: Idempotent
    

    View Slide

79. View Slide

80. Halo Statistics
    

    View Slide

81. Halo Statistics Service
    

    View Slide

82. Halo Statistics Service with Sagas
    

    View Slide

83. Forward-Recovery
    

    View Slide

84. Store Stats
    Player One
    Store Stats
    Player Two
    Store Stats
    Player Three
    Store Stats
    Player Four
    Game Grain / SEC
    

    View Slide

85. Forward Recovery
    Sub-Requests Must Also Be
    Idempotent
    

    View Slide

86. Store Stats
    Player One
    Store Stats
    Player Two
    Store Stats
    Player Three
    Store Stats
    Player Four
    Game Grain / SEC
    

    View Slide

87. Store Stats
    Player One
    Store Stats
    Player Two
    Store Stats
    Player Three
    Store Stats
    Player Four
    Game Grain / SEC
    

    View Slide

88. Sagas
    • Long Lived / Distributed Transactions
    • Trade Atomicity for Availability
    • Failure Management Pattern
    

    View Slide

89. Thank You
    • @pbailis
    • @randommood
    • @tsantero
    • @aphyr
    • @jmhodges
    • @clemnsv
    

    View Slide

90. Questions?
    Please remember to evaluate via the GOTO Guide App
    @Caitie
    

    View Slide