Distributed Sagas
A Protocol for Coordinating Microservices
Slide 2
Slide 2 text
Caitie McCaffrey
Distributed Systems Engineer
caitiem.com
@caitie
Slide 3
Slide 3 text
Service Service
Service
Monoliths
Slide 4
Slide 4 text
Microservices & NoSQL
Slide 5
Slide 5 text
2015
Slide 6
Slide 6 text
2015
“Application-level
Mechanisms for maintaining
database integrity”
Slide 7
Slide 7 text
Reserve a Hotel
Front End
Slide 8
Slide 8 text
Reserve a Hotel
Front End
Feral Concurrency
Control Mechanisms
Slide 9
Slide 9 text
Reserve a Hotel
Front End
Feral Concurrency
Control Mechanisms
Slide 10
Slide 10 text
Reserve a Car
Front End
Slide 11
Slide 11 text
Reserve a Flight
Front End
Slide 12
Slide 12 text
Trips
Front End
Slide 13
Slide 13 text
Trips
Front End
Trips
Slide 14
Slide 14 text
Trips
Front End
Trips
Feral Concurrency
Control Mechanisms
Slide 15
Slide 15 text
Trips
Front End
Trips
Feral Concurrency
Control Mechanisms
Slide 16
Slide 16 text
Trips
Front End
Trips
Feral Concurrency
Control Mechanisms
Slide 17
Slide 17 text
Front End
Trips
Architecture
Slide 18
Slide 18 text
Death Star Architectures
Slide 19
Slide 19 text
Can We Do Better
than Feral Concurrency Control?
Slide 20
Slide 20 text
2012
Slide 21
Slide 21 text
2012
“Spanner is Google’s scalable, multi-
version, globally distributed, and
synchronously-replicated database”
Slide 22
Slide 22 text
2015
Slide 23
Slide 23 text
2015
“The biggest barrier…is that
consistency mechanisms must
integrate across many
stateful services”
Slide 24
Slide 24 text
Two Phase Commit
Slide 25
Slide 25 text
2PC: Prepare Front End
Trips
Propose Book Hotel
Propose Book Car
Propose Book
Flight
Slide 26
Slide 26 text
2PC: Prepare Front End
Trips
Vote Book Hotel
Vote Book Car
Vote Book
Flight
Slide 27
Slide 27 text
2PC: Commit Front End
Trips
Commit/Abort Book Hotel
Commit/Abort Book Car
Commit/Abort
Book Flight
Slide 28
Slide 28 text
2PC: Commit Front End
Trips
Done
Done
Done
Slide 29
Slide 29 text
2PC: Commit Front End
Trips
Done
Done
Done
Doesn’t Scale
•O (N^2) Messages in the worst case
•Coordinator is a Single Point of Failure
•Reduced Throughput
Slide 30
Slide 30 text
Distributed Sagas
A Protocol for Coordinating Microservices
Slide 31
Slide 31 text
1987
Slide 32
Slide 32 text
1987
“Sagas are Long Lived Transactions
[in a single relational Database]”
Slide 33
Slide 33 text
1987
“A Saga is a Long Lived Transaction that
can be written as a sequence of transactions
that can be interleaved.
All transactions in the sequence complete
successfully or compensating transactions
are ran to amend a partial execution.”
Slide 34
Slide 34 text
Distributed Sagas
A Protocol for Coordinating Microservices
Slide 35
Slide 35 text
A Distributed Saga is a Collection of Requests
Book Hotel Book Car Book Flight Charge Money
Slide 36
Slide 36 text
A Distributed Saga is a Collection of Requests
Book Hotel Book Car Book Flight Charge Money
and Compensating Requests
Cancel Hotel Cancel Car Cancel Flight Refund Money
Slide 37
Slide 37 text
A Distributed Saga is a Collection of Requests
Book Hotel Book Car Book Flight Charge Money
and Compensating Requests
Cancel Hotel Cancel Car Cancel Flight Refund Money
that represent a single business level action
Slide 38
Slide 38 text
Distributed Saga
Requests
Slide 39
Slide 39 text
Requests
Can Abort
Book Car
Slide 40
Slide 40 text
Requests
Can Abort
Book Car
Slide 41
Slide 41 text
Requests
Must Be Idempotent
Book Car
Slide 42
Slide 42 text
Requests
Must Be Idempotent
Book Car
Slide 43
Slide 43 text
Requests
Must Be Idempotent
Book Car
Slide 44
Slide 44 text
Requests
Must Be Idempotent
Book Car
Slide 45
Slide 45 text
Requests
Must Be Idempotent
Book Car
Slide 46
Slide 46 text
Requests
Must Be Idempotent
Book Car
Timeout
Slide 47
Slide 47 text
Requests
Must Be Idempotent
Book Car
Slide 48
Slide 48 text
Requests
Must Be Idempotent
Book Car
Slide 49
Slide 49 text
Distributed Saga
Compensating Requests
Slide 50
Slide 50 text
Compensating Requests
Semantically undoes the effect of a request
Slide 51
Slide 51 text
Cancel Car
Compensating
Requests
Cannot Abort
Slide 52
Slide 52 text
Cancel Car
Compensating
Requests
Can Not Abort
Slide 53
Slide 53 text
Cancel Car
Compensating
Requests
Must Be Idempotent
Slide 54
Slide 54 text
Compensating
Requests
Must Be Commutative
with Requests
Book Car
Cancel Car
Cancel Car
Book Car
is the same as
Slide 55
Slide 55 text
Compensating
Requests
Book Car
Must Be Commutative
with Requests
Slide 56
Slide 56 text
Compensating
Requests
Book Car
Timeout
Must Be Commutative
with Requests
Slide 57
Slide 57 text
Compensating
Requests
Book Car
Book Car
Must Be Commutative
with Requests
Slide 58
Slide 58 text
Compensating
Requests
Book Car
Book Car
Cancel Car
Must Be Commutative
with Requests
Slide 59
Slide 59 text
Compensating
Requests
Book Car
Book Car
Cancel Car
Must Be Commutative
with Requests
Slide 60
Slide 60 text
Compensating
Requests
Requests
Idempotent Idempotent
Commutative
Can Not Abort
Can Abort
Slide 61
Slide 61 text
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Slide 62
Slide 62 text
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding
compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
Slide 63
Slide 63 text
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding
compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
No Atomicity
No Isolation
Slide 64
Slide 64 text
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding
compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
No Atomicity
No Isolation
Book Hotel
Book Car
Visible before
Saga Completes
Slide 65
Slide 65 text
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding
compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
Slide 66
Slide 66 text
Distributed Saga
defining a
Slide 67
Slide 67 text
Directed Acyclic Graph
Distributed Saga Car Flight
Hotel
Payment
Start Saga
End Saga
Slide 68
Slide 68 text
Name: Hotel
Request: Book Hotel
Compensating Request: Cancel Hotel
Status: Not Completed
Vertex
Distributed Saga
Slide 69
Slide 69 text
Start & End Vertices
Distributed Saga
Start Saga
End Saga
Car Flight
Hotel
Payment
Start Saga
End Saga
Slide 70
Slide 70 text
Distributed Saga Log
fault-tolerant & highly available
Slide 71
Slide 71 text
Saga Execution Coordinator
SEC
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Slide 72
Slide 72 text
Distributed Saga
Executing a
Slide 73
Slide 73 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Book Trip Request
SEC
Slide 74
Slide 74 text
Car Flight
Hotel
Payment
Start Saga
End Saga
SEC
Saga Log
Start Saga
Book Trip Request
Slide 75
Slide 75 text
Car Flight
Hotel
Payment
Start Saga
End Saga
SEC
Saga Log
Done
Start Saga
Book Trip Request
Slide 76
Slide 76 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
SEC
Start Hotel
Start Saga
Slide 77
Slide 77 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
SEC
Done
Start Saga
Start Hotel
Slide 78
Slide 78 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
SEC
Book Hotel
Request
Start Saga
Start Hotel
Slide 79
Slide 79 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
SEC
Book Hotel
Response
Start Saga
Start Hotel
Slide 80
Slide 80 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
SEC
End Hotel
Start Saga
Start Hotel
Slide 81
Slide 81 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
SEC
Done
Start Saga
Start Hotel
End Hotel
Slide 82
Slide 82 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
SEC
Start Car
Start Saga
Start Hotel
End Hotel
Slide 83
Slide 83 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
SEC
Done
Book Car
Request
Start Saga
Start Hotel
End Hotel
Start Car
Slide 84
Slide 84 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Book Car
Response
End Car
Slide 85
Slide 85 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Done
End Car
Slide 86
Slide 86 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
End Car
Slide 87
Slide 87 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Done
End Car
Start Flight Book Flight
Request
Slide 88
Slide 88 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
Book Flight
Response
End Flight
Slide 89
Slide 89 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
Done
End Flight
Slide 90
Slide 90 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
Start Payment
End Flight
Slide 91
Slide 91 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
Done
End Flight
Start Payment
Payment
Request
Slide 92
Slide 92 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Payment
End Flight
Start Payment
Payment
Response
Slide 93
Slide 93 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Flight
Start Payment
End Payment
Done
Slide 94
Slide 94 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Flight
Start Payment
End Payment
End Saga
Slide 95
Slide 95 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Flight
Start Payment
End Payment
Done
End Saga
Slide 96
Slide 96 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
End Car
Start Flight
End Flight
Start Payment
End Payment
End Saga
Slide 97
Slide 97 text
Distributed Saga
Failure of a
Slide 98
Slide 98 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
SEC
Book Car
Request
Start Saga
Start Hotel
End Hotel
Start Car
Start Flight
Slide 99
Slide 99 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Book Car
Response
Abort Car
Start Car
Start Flight
Slide 100
Slide 100 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Done
Start Car
Start Flight
Abort Car
Slide 101
Slide 101 text
Car Flight
Hotel
Payment
Start Saga
End Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Done
Start Car
Start Flight
Abort Car
Rollback Recovery
Slide 102
Slide 102 text
Car Flight
Hotel
Payment
End Comp Saga
Start Comp Saga
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Slide 103
Slide 103 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Start Comp Saga
End Comp Saga
Slide 104
Slide 104 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Payment Log
Entries?
Start Car
Start Flight
Abort Car
Start Comp Saga
End Comp Saga
Slide 105
Slide 105 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
{ }
Start Car
Start Flight
Abort Car
Start Comp Saga
End Comp Saga
Slide 106
Slide 106 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Car Log
Entries?
Start Comp Saga
End Comp Saga
Slide 107
Slide 107 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
{ Start, Abort }
Start Comp Saga
End Comp Saga
Slide 108
Slide 108 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Hotel Log
Entries?
Start Comp Saga
End Comp Saga
Slide 109
Slide 109 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
{ Start, End }
Start Comp Saga
End Comp Saga
Slide 110
Slide 110 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
{ Start, End }
Cancel Hotel
Request
Start Comp Saga
End Comp Saga
Slide 111
Slide 111 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Cancel Hotel
Response
Comp Hotel
Start Comp Saga
End Comp Saga
Slide 112
Slide 112 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Done
Comp Hotel
Start Comp Saga
End Comp Saga
Slide 113
Slide 113 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Flight Log
Entries?
Start Comp Saga
End Comp Saga
Slide 114
Slide 114 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
{ Start }
Start Comp Saga
End Comp Saga
Slide 115
Slide 115 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
{ Start }
Book Flight
Request
Start Comp Saga
End Comp Saga
Slide 116
Slide 116 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
End Flight
Book Flight
Response
Start Comp Saga
End Comp Saga
Slide 117
Slide 117 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Done
Cancel Flight
Request
End Flight
Start Comp Saga
End Comp Saga
Slide 118
Slide 118 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Comp Flight
Cancel Flight
Response
Start Comp Saga
End Comp Saga
End Flight
Slide 119
Slide 119 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Done
Comp Flight
Start Comp Saga
End Comp Saga
End Flight
Comp Flight
Slide 120
Slide 120 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
End Saga
Start Comp Saga
End Comp Saga
Comp Flight
End Flight
Comp Flight
Slide 121
Slide 121 text
Car Flight
Hotel
Payment
Saga Log
Start Saga
SEC
Start Hotel
End Hotel
Start Car
Start Flight
Abort Car
Comp Hotel
Done
End Saga
Start Comp Saga
End Comp Saga
Comp Flight
End Flight
Comp Flight
Slide 122
Slide 122 text
Distributed Saga Guarantee
Book Hotel Book Car Book Flight Charge Money
All requests were completed successfully
Or a subset of requests and the corresponding
compensating requests were executed
Book Hotel Book Car Cancel Hotel Cancel Car
Slide 123
Slide 123 text
Saga Execution Coordinator
Recovering from
Failure
Slide 124
Slide 124 text
Front End
SEC
Start Saga
End Saga
Saga Log
Slide 125
Slide 125 text
Front End
SEC
Start Saga
End Saga
Saga Log
Slide 126
Slide 126 text
Front End
SEC
Start Saga
End Saga
Saga Log
Slide 127
Slide 127 text
Front End
SEC
Saga Log
Start Saga
End Saga
Slide 128
Slide 128 text
with Distributed Sagas
Isolation of Complex Code
Fron
Tr
Complex Code
Lives Everywhere
Front
SEC
Saga Log
Complex Code
Lives Here
Complex Code
Lives Here
Slide 129
Slide 129 text
Fron
Tr
with Distributed Sagas
Modular Services
Front
SEC
Saga Log
Only Flight
Booking
Flight Booking &
Feral Concurrency
Control Mechanisms
Slide 130
Slide 130 text
Fron
Tr
with Distributed Sagas
Service Composition
Front
SEC
Saga Log
Just Add A
New Saga! New Service &
Feral Concurrency
Control Mechanisms
Slide 131
Slide 131 text
Distributed Sagas Makes
Building & Modifying
Microservices Easier