Designing Events-first Microservices

In this talk, we will explore the nature of events, what it means to be event-driven, and how we can unleash the power of events and commands by applying an events-first domain-driven design to microservices-based architectures.

We will start by developing a solid theoretical understanding of how to design systems of event-driven microservices. Then we will discuss the practical tools and techniques you can use to reap the most benefit from that design, as well as, most importantly, what to avoid along the way.

We’ll discuss how an events-first design approach to building microservices can improve the following characteristics over competing techniques:
- increase certainty
- increase resilience
- increase scalability
- increase traceability
- increase loose coupling
- reduce risk

Skeptics should definitely attend.

Transcript

1. Designing Events First Microservices Jonas Bonér @jboner
2. 1. Events drive autonomy 2. Events help reduce risk 3. Events help you move faster 4. Events Increase Loose coupling 5. Events increase stability 6. Events increase scalability 7. Events increase resilience 8. Events increase traceability 9. Events allow for time-travel Why Should you care about Events?
3. Why Now? 1. Cloud and multicore architectures 2. Microservices and distributed systems 3. Data-centric applications - Data in motion 4. “We want more, of everything, and we want it now.” -Your Customers
4. So, you want to do microservices?
5. Make sure you don’t end up with Microliths
6. Make sure you don’t end up with Microliths We can do better than this
7. Events First Domain Driven Design
8. “When you start modeling events, it forces you to think about the behaviour of the system. As opposed to thinking about the structure of the system.” - Greg Young A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
9. ✴ Don’t focus on the things The Nouns The Domain Objects
10. ✴ Don’t focus on the things The Nouns The Domain Objects ✴ Focus on what happens The Verbs The Events
11. What is an Event?
12. The Nature of Events
13. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow The Nature of Events
14. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored The Nature of Events
15. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) The Nature of Events
16. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) ✴ Events/Facts Can not be deleted (once accepted) ➡ Might be needed for legal or moral reasons The Nature of Events
17. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) ✴ Events/Facts Can not be deleted (once accepted) ➡ Might be needed for legal or moral reasons ✴ Events/Facts (new) can invalidate existing Facts The Nature of Events
18. Mine the Facts
20. Event Storming
21. Event Driven Design
22. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts ➡ Control Transfer Event Driven Design
23. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer
24. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer Commands
25. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer Commands Events
26. Event Driven Design
27. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder, ShipProduct Event Driven Design
28. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder, ShipProduct ✴Reactions ➡ Represents side-effects Event Driven Design
29. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder, ShipProduct ✴Reactions ➡ Represents side-effects ✴Events ➡ Represents something that has happened ➡ Past-tense: OrderCreated, ProductShipped Event Driven Design
30. Commands Events vs
31. Commands Events vs 1. All about intent 1. Intentless
32. Commands Events vs 1. All about intent 2. Directed 1. Intentless 2. Anonymous
33. Commands Events vs 1. All about intent 2. Directed 3. Single addressable destination 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe
34. Commands Events vs 1. All about intent 2. Directed 3. Single addressable destination 4. Models personal communication 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner)
35. Commands Events vs 1. All about intent 2. Directed 3. Single addressable destination 4. Models personal communication 5. Distributed focus 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner) 5. Local focus
36. Commands Events vs 1. All about intent 2. Directed 3. Single addressable destination 4. Models personal communication 5. Distributed focus 6. Command & Control 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner) 5. Local focus 6. Autonomy
37. Let the Events Define the Bounded Context
38. Event Driven Services
39. 1. REceive & react (or not) 
 to facts* that are coming its way Event Driven Services
40. 1. REceive & react (or not) 
 to facts* that are coming its way 2. Publish new facts* asynchronously 
 to the rest of the world Event Driven Services
41. 1. REceive & react (or not) 
 to facts* that are coming its way 2. Publish new facts* asynchronously 
 to the rest of the world 3. Invert the control flow
 to minimize coupling and increase autonomy Event Driven Services
42. 1. REceive & react (or not) 
 to facts* that are coming its way 2. Publish new facts* asynchronously 
 to the rest of the world 3. Invert the control flow
 to minimize coupling and increase autonomy Event Driven Services *Facts == Immutable Events
43. Mutable State Is Fine But Needs To Be Contained And Non Observable
44. Publish Facts To Outside World
45. ✴ Maintains Integrity & consistency ✴ Is our Unit of Consistency ✴ Is our Unit of Failure ✴ Is our Unit of Determinism ✴ Is fully Autonomous The Aggregate
46. Event Driven Services
47. Event Driven Services
48. Event Driven Services Command
49. Event Driven Services Command
50. Event Driven Services Command
51. Event Driven Services Command Event Event Stream
52. Event Driven Services Command Event Event Event Event Stream
53. Event Driven Services Command Event Event Event Event Stream
54. Event Driven Services Command Event Event Event Event Stream
55. Event Driven Services Command Event Event Event Event Stream
56. Event Driven Services Command Event Event Event Event Stream
57. Event Driven Services Eventual Consistency Command Event Event Event Event Stream
58. Event Stream Use The
59. Event Stream Use The as the communication fabric
60. Event Stream Use The
61. Event Stream Use The as the integration fabric
62. Event Stream Use The
63. Event Stream Use The as the replication fabric
64. Event Stream Use The
65. Event Stream Use The as the consensus fabric
66. Event Stream Use The
67. Event Stream Use The as the Persistence fabric
68. The Problem With CRUD Services
69. ✴ CRUD is fine for totally isolated data The Problem With CRUD Services
70. ✴ CRUD is fine for totally isolated data ✴ But, cross CRUD services consistency The Problem With CRUD Services
71. ✴ CRUD is fine for totally isolated data ✴ But, cross CRUD services consistency ➡ Is hard ⇨ No Joins The Problem With CRUD Services
72. ✴ CRUD is fine for totally isolated data ✴ But, cross CRUD services consistency ➡ Is hard ⇨ No Joins ➡ Has ad-hoc & weak guarantees The Problem With CRUD Services
73. ✴ CRUD is fine for totally isolated data ✴ But, cross CRUD services consistency ➡ Is hard ⇨ No Joins ➡ Has ad-hoc & weak guarantees The Problem With CRUD Services
74. “In general, application developers simply do not implement large scalable applications assuming distributed transactions” - Pat Helland Life Beyond Distributed Transactions - Pat Helland
75. “Two-phase commit is the anti-availability protocol.” - Pat Helland Standing on Distributed Shoulders of Giants - Pat Helland
76. STRONG Consistency Is the wrong default In distributed systems
77. STRONG Consistency Is the wrong default In distributed systems What can we do?
78. Eventual Consistency We have to rely on
79. Eventual Consistency We have to rely on But relax—it’s how the world works
80. Embrace Reality We Need to
81. Embrace Reality We Need to Not Fight it
82. Information Has Latency
83. Information Is Always From the Past
84. Welcome To The Wild Ocean Of Non Determinism Distributed Systems
85. “In a system which cannot count on distributed transactions, the management of uncertainty must be implemented in the business logic.” - Pat Helland Life Beyond Distributed Transactions, Pat Helland (2007) We Need To Model Uncertainty
86. Events Can Lead To Greater Certainty
87. “An autonomus component can only promise its own behavior.” “Autonomy makes information local, leading to greater certainty and stability.” - Mark Burgess In Search of Certainty, Thinking in Promises - Mark Burgess
88. Events Can Help Us Craft Autonomous Islands Of Determinism
89. Data on the inside vs Data on the outside - Pat Helland
90. Data on the inside vs Data on the outside - Pat Helland Inside Data Our current present ⇨ state
91. Data on the inside vs Data on the outside - Pat Helland Inside Data Our current present ⇨ state Outside Data Blast from the past ⇨ Events/facts
92. Data on the inside vs Data on the outside - Pat Helland Inside Data Our current present ⇨ state Outside Data Blast from the past ⇨ Events/facts Between Services Hope for the future ⇨ commands
93. A system of microservices is a never ending stream towards convergence
94. There Is No Now A system of microservices is a never ending stream towards convergence
95. Resilience is by Design Photo courtesy of FEMA/Joselyne Augustino
96. Events Can Help Us Manage Failure Instead Of Trying To Avoid It
97. Requirements for a Sane Failure Model 1. Contained—Avoid cascading failures 2. Reified—as Events 3. Signalled—Asynchronously 4. Observed—by 1-N 5. Managed—Outside failed Context Failures need to be
98. Event Based Persistence
99. You can use CRUD Together with Event Streams To get an internally consistent Materialized View
100. Service B Service A You can use CRUD Together with Event Streams To get an internally consistent Materialized View
101. Service B Service A CRUD You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD
102. Service B Service A TABLE A CRUD TABLE B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD
103. Service B Service A TABLE A CRUD TABLE B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD
104. Service B Service A TABLE A CRUD TABLE B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
105. Service B Service A TABLE A CRUD TABLE B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
106. Service C Service B Service A TABLE A CRUD TABLE B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
107. Service C Service B Service A TABLE A CRUD TABLE B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
108. Service C Service B Service A TABLE A CRUD TABLE B JOINS Table A Table B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
109. Service C Service B Service A TABLE A CRUD TABLE B JOINS Table A Table B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Read Only Atomic Update & event
110. Service C Service B Service A Eventual Consistency TABLE A CRUD TABLE B JOINS Table A Table B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Read Only Atomic Update & event
111. “Update-in-place strikes systems designers as a cardinal sin: it violates traditional accounting practices that have been observed for hundreds of years.” - jim Gray The Transaction Concept, Jim Gray (1981)
112. “The truth is the log. The database is a cache of a subset of the log.” - Pat Helland Immutability Changes Everything, Pat Helland (2015)
113. Event Logging The Bedrock
114. Event Sourcing A Cure For the Cardinal Sin
115. Event Sourced Services
116. Happy Path Event Sourced Services
117. Happy Path Event Sourced Services 1) Receive and verify Command (“ApprovePayment”)
118. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”)
119. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log
120. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
121. Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
122. SAD Path - recover from failure Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
123. SAD Path - recover from failure Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log
124. SAD Path - recover from failure Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log 2) Update internal component state
125. SAD Path - recover from failure Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log 2) Update internal component state Memory Image
126. Event Sourcing
127. Event Sourcing ✴ One single Source of Truth with All history
128. Event Sourcing ✴ One single Source of Truth with All history ✴ Allows for Memory Image (Durable In-Memory State)
129. Event Sourcing ✴ One single Source of Truth with All history ✴ Allows for Memory Image (Durable In-Memory State) ✴ Avoids the Object-relational mismatch
130. Event Sourcing ✴ One single Source of Truth with All history ✴ Allows for Memory Image (Durable In-Memory State) ✴ Avoids the Object-relational mismatch ✴ Allows others to Subscribe to state changes
131. Event Sourcing ✴ One single Source of Truth with All history ✴ Allows for Memory Image (Durable In-Memory State) ✴ Avoids the Object-relational mismatch ✴ Allows others to Subscribe to state changes ✴ Has good Mechanical sympathy
 (Single Writer Principle etc.)
132. ✴ Unfamiliar model ✴ Versioning of events ✴ Deletion of events (legal or moral reasons) Disadvantages Of Using Event Sourcing
133. Untangle Your Read and Write Models With CQRS
134. CQRS
135. CQRS
136. CQRS Write Side Model Commands
137. CQRS Write Side Model Write to Event Log Commands
138. CQRS Write Side Model Events Write to Event Log Commands
139. CQRS Write Side Model Events Events Write to Event Log Commands
140. CQRS Read Side Model Write Side Model Events Events Read Data Store Write to Event Log Commands
141. CQRS Read Side Model Write Side Model Events Queries Events Read Data Store Write to Event Log Commands
142. CQRS Read Side Model Write Side Model Events Queries Events Read Data Store Write to Event Log Eventual Consistency Commands
143. ✴ Temporal Decoupling ✴ Increased Resilience ✴ Increased Scalability ✴ Allows for Multiple Event subscribers ✴ Allows for Polyglot Persistence Advantages Of Using CQRS
144. ✴ More infrastructure to manage ✴ 2 data models (or more) to maintain ✴ Eventual Consistency Disadvantages Of Using CQRS
145. Events Allow Us To Manage Time
146. “Modelling events forces you to have a 
 temporal focus on what’s going on in the system. Time becomes a crucial factor of the system.” - Greg Young A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
147. ✴ Event is a snapshot in time ✴ Event ID is an index for time ✴ Event Log is our full history The database of Our past The path to Our present Event Sourcing Allows Us To Model Time
148. Event Sourcing Allows For Time Travel
149. Event Sourcing Allows For Time Travel
150. Event Sourcing Allows For Time Travel ✴Replay the log for historic debugging ✴Replay the log for auditing & traceability ✴Replay the log on failure ✴Replay the log for replication
151. We Can Even Fork the Past ...Or Join Two Distinct Pasts
152. Key Takeaways
153. Key Takeaways Events-First design helps you to: ✴ reduce risk when modernizing applications ✴ Move Faster towards a Resilient and Scalable architecture ✴ Design autonomous services ✴ Balance Certainty and Uncertainty
154. Key Takeaways Events-First design helps you to: ✴ reduce risk when modernizing applications ✴ Move Faster towards a Resilient and Scalable architecture ✴ Design autonomous services ✴ Balance Certainty and Uncertainty Event Logging allows you to: ✴ AVOID CRUD and ORM ✴ take control of your system’s history ✴ time-travel ✴ Balance Strong and eventual consistency
155. http://akka.io
156. Learn More Download my latest book for free at: bit.ly/reactive-microsystems