How Events Are Reshaping Modern Systems

Event-driven architecture and design have been getting a lot of attention in recent years. It’s an old concept that has been around for decades, so why this sudden peak of interest? 

In this talk, we will explore the nature of events, what it means to be event-driven, and how we can unleash the power of events. The goal is to arm you with a solid theoretical understanding of how to design an event-driven system, what tools and techniques you can use to reap the most benefit from its design, and perhaps most importantly, what to avoid.

We'll discuss how an event-driven design can help: 
- drive autonomy
- reduce risk
- increase certainty
- increase loose coupling
- increase scalability
- increase resilience
- increase traceability

Skeptics should definitely attend.

Transcript

1. How Events Are Reshaping Modern Systems Jonas Bonér @jboner
2. Why Events?
3. 1. Events drive autonomy 2. Events help reduce risk 3. Events help you move faster 4. Events Increase Loose coupling 5. Events increase stability 6. Events increase scalability 7. Events increase resilience 8. Events increase traceability 9. Events allow for time-travel Why Should you care?
4. Why Now? 1. Cloud and multicore architectures 2. Microservices and distributed systems 3. Data-centric applications 4. “We want more, of everything, and we want it now.” - Your Customers
5. Do Not Settle For Microliths
6. Do Not Settle For Microliths
7. “Event-Driven Architecture (EDA) is a design paradigm in which a software component executes in response to receiving one or more event notifications.” - Gartner Event Driven Architecture To The Rescue
8. What is an Event?
9. Event “Something that happens.” - Merriam Webster
10. The Nature of Events
11. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow The Nature of Events
12. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored The Nature of Events
13. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) The Nature of Events
14. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) ✴ Events/Facts Can not be deleted (once accepted) ➡ Might be needed for legal or moral reasons The Nature of Events
15. ✴ Events represent Facts of information ➡ Facts are immutable ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) ✴ Events/Facts Can not be deleted (once accepted) ➡ Might be needed for legal or moral reasons ✴ Events/Facts (new) can invalidate existing Facts The Nature of Events
16. Event Loop What Makes Event Tick
17. ✴ Queue + anonymous Callbacks ✴ The Hollywood principle ✴ At-Most-Once delivery guarantees ✴ Local context only ✴ Sync or AsynC ✴ Limited and quite powerless The Event Loop
18. ✴ Queue + anonymous Callbacks ✴ The Hollywood principle ✴ At-Most-Once delivery guarantees ✴ Local context only ✴ Sync or AsynC ✴ Limited and quite powerless The Event Loop * * Too limited as a general model of computation, and programming model for modern distributed systems. But is a useful building block.
19. The Problem With Event Callbacks
20. ✴ Ephemeral and anonymous The Problem With Event Callbacks
21. ✴ Ephemeral and anonymous ✴ No sane failure model Error Channel - console.log(‘4/0=err’, err) The Problem With Event Callbacks
22. ✴ Ephemeral and anonymous ✴ No sane failure model Error Channel - console.log(‘4/0=err’, err) ✴ No composition Signature: Input => Side-effect - (Any => Unit) The Problem With Event Callbacks
23. ✴ Ephemeral and anonymous ✴ No sane failure model Error Channel - console.log(‘4/0=err’, err) ✴ No composition Signature: Input => Side-effect - (Any => Unit) ✴ Hard to Debug The Problem With Event Callbacks
24. ✴ Ephemeral and anonymous ✴ No sane failure model Error Channel - console.log(‘4/0=err’, err) ✴ No composition Signature: Input => Side-effect - (Any => Unit) ✴ Hard to Debug ✴ Hard to reason about The Problem With Event Callbacks
25. ✴ Ephemeral and anonymous ✴ No sane failure model Error Channel - console.log(‘4/0=err’, err) ✴ No composition Signature: Input => Side-effect - (Any => Unit) ✴ Hard to Debug ✴ Hard to reason about ✴ Leads to Callback hell The Problem With Event Callbacks
26. https://techbrunch.gousto.co.uk/2016/04/22/callback-hell/
28. ✴ Patterns Observer, Pub-Sub, Broadcast, … ✴ Futures & PRomises ✴ Dataflow Variables ✴ Stream Processing ✴ Event-driven Microservices ✴ FaaS We Need High Level Abstractions
29. Unified Event Driven Abstractions Powered By Message Driven Fabric Ladder of abstraction
30. Unified Event Driven Abstractions Powered By Message Driven Fabric Network Boundary Ladder of abstraction
31. Unified Event Driven Abstractions Powered By Message Driven Fabric Network Boundary Ladder of abstraction
32. Unified Event Driven Abstractions Powered By Message Driven Fabric Network Boundary Ladder of abstraction
33. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Network Boundary Ladder of abstraction
34. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Network Boundary Event 1 Ladder of abstraction
35. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Network Boundary Event 1 Ladder of abstraction
36. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Network Boundary Event 1 Ladder of abstraction
37. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Distribution Fabric Network Boundary Message Passing Event 1 Ladder of abstraction
38. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Ladder of abstraction
39. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Ladder of abstraction
40. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Event 1 Ladder of abstraction
41. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Event 1 Reactive Systems is doing the heavy lifting Ladder of abstraction
42. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Event 1 Reactive Systems is doing the heavy lifting Ladder of abstraction Reactive Programming
43. 1. REceive and react to facts (or not), that are coming its way 2. Publish new facts (as events) to the rest of the world 3. Invert the control flow to minimize coupling and increase autonomy Event Driven Services
44. Mutable State Needs To Be Contained And Non Observable
45. Publish Facts To Outside World
46. Event Stream
47. Event Stream Use The as the integration fabric
48. Event Driven Services
49. Event Driven Services
50. Event Driven Services Command
51. Event Driven Services Command
52. Event Driven Services Command
53. Event Driven Services Command Event Event Stream
54. Event Driven Services Command Event Event Event Event Stream
55. Event Driven Services Command Event Event Event Event Stream
56. Event Driven Services Command Event Event Event Event Stream
57. Event Driven Services Command Event Event Event Event Stream
58. Event Driven Services Command Event Event Event Event Stream
59. Event Driven Services Eventual Consistency Command Event Event Event Event Stream
60. Event Driven Services Eventual Consistency Command Event Event Event Event Stream
61. Eventual Consistency
62. Eventual Consistency No One Wants It’s a necessery evil
63. But Relax It Is How The World Works
64. Information Has Latency
65. Information Is Always From the Past
66. Welcome To The Wild Ocean Of Non Determinism Distributed Systems
67. “In a system which cannot count on distributed transactions, the management of uncertainty must be implemented in the business logic.” - Pat Helland Life Beyond Distributed Transactions - An Apostate’s Opinion, Pat Helland (2007) We Need To Model Uncertainty
68. Exploit Reality
69. Exploit Reality We need to In our design
70. Events Can Lead To Greater Certainty
71. Events Can Help Us Craft Autonomous Islands Of Determinism
73. What does it mean to be Automonous?
74. From greek: Auto-nomos ✴ Auto meaning self ✴ Nomos meaning Law Autonomy
75. Promise Theory
76. Promise Theory Let Lead the way
77. ✴Impositions diverge into unpredictable outcomes from definite beginnings 㱺 Distributed information 㱺 Decreased certainty Convergence vs. Divergence of information ✴Promises converge towards a definite outcome from unpredictable beginnings 㱺 Local information 㱺 Improved certainty
78. “Autonomy makes information local, leading to greater certainty and stability.” “An autonomus component can only promise its own behavior.” - Mark Burgess In Search of Certainty, Thinking in Promises - Mark Burgess
79. Benefits of Thinking in Promises
80. ✴ If a service only promises its own behavior Then all information needed to ➡ Resolve a conflict ➡ Repair under failure ➡ Is available within the service itself Benefits of Thinking in Promises
81. ✴ If a service only promises its own behavior Then all information needed to ➡ Resolve a conflict ➡ Repair under failure ➡ Is available within the service itself ✴ Promises remove the need for needless 1. Communication 2. Coordination 3. Consensus Benefits of Thinking in Promises
82. “The first principle of successful scalability is to batter the consistency mechanisms down to a minimum.” - James Hamilton As transcribed in: Toward a Cloud Computing Research Agenda. SIGACT News, 40(2):68–80, June 2009.
83. Events First Design Drives Autonomy
84. Events First Domain Driven Design
85. “When you start modeling events, it forces you to think about the behavior of the system. As opposed to thinking about the structure of the system.” - Greg Young A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
86. ✴ Don’t focus on the things The Nouns The Domain Objects ✴ Focus on what happens The Verbs The Events
87. Mine the Facts
89. Event Storming
90. Event Driven Design
91. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts ➡ Control Transfer Event Driven Design
92. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer
93. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer Commands
94. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer Commands Events
95. Event Driven Design
96. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder, ShipProduct Event Driven Design
97. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder, ShipProduct ✴Reactions ➡ Represents side-effects Event Driven Design
98. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder, ShipProduct ✴Reactions ➡ Represents side-effects ✴Events ➡ Represents something that has happened ➡ Past-tense: OrderCreated, ProductShipped Event Driven Design
99. Commands Events vs
100. Commands Events vs 1. All about intent 1. Intentless
101. Commands Events vs 1. All about intent 2. Directed 1. Intentless 2. Anonymous
102. Commands Events vs 1. All about intent 2. Directed 3. Single addressable destination 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe
103. Commands Events vs 1. All about intent 2. Directed 3. Single addressable destination 4. Models personal communication 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner)
104. Commands Events vs 1. All about intent 2. Directed 3. Single addressable destination 4. Models personal communication 5. Distributed focus 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner) 5. Local focus
105. Commands Events vs 1. All about intent 2. Directed 3. Single addressable destination 4. Models personal communication 5. Distributed focus 6. Command & Control 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner) 5. Local focus 6. Autonomy
106. Let the Events Define the Bounded Context
107. Principles For Promise Theory Driven Protocol Design
108. 1. Convergency Promises should reach a desired state, a stable outcome - idempotent behavior Principles For Promise Theory Driven Protocol Design
109. 1. Convergency Promises should reach a desired state, a stable outcome - idempotent behavior 2. Embracing Failure Expect things to go wrong Promises may or may not be kept Principles For Promise Theory Driven Protocol Design
110. 1. Convergency Promises should reach a desired state, a stable outcome - idempotent behavior 2. Embracing Failure Expect things to go wrong Promises may or may not be kept 3. Autonomy You can’t rely on other services Principles For Promise Theory Driven Protocol Design
111. ✴ Maintains Integrity & consistency ✴ Is our Unit of Consistency ✴ Is our Unit of Failure ✴ Is our Unit of Determinism ✴ Is fully Autonomous The Aggregate
112. Enough Talk Show Me The Code Sample in Java: https://gist.github.com/jboner/4361c710d2e2386be8bddc39edb0528a Sample in Scala: https://gist.github.com/jboner/dea4aa819e127d543d684208d74081b5
113. Enough Talk Show Me The Code Sample in Java: https://gist.github.com/jboner/4361c710d2e2386be8bddc39edb0528a Sample in Scala: https://gist.github.com/jboner/dea4aa819e127d543d684208d74081b5
114. Are we done now? Perhaps. We have come a long way. But events can also be used for: ➡ Persistence ➡ Managing time
115. Event Based Persistence
116. “Update-in-place strikes systems designers as a cardinal sin: it violates traditional accounting practices that have been observed for hundreds of years.” - jim Gray The Transaction Concept, Jim Gray (1981)
117. Event Logging The Bedrock
118. Event Sourcing A Cure For the Cardinal Sin
119. “The truth is the log. The database is a cache of a subset of the log.” - Pat Helland Immutability Changes Everything, Pat Helland (2015)
120. Event Sourced Services
121. Happy Path Event Sourced Services
122. Happy Path Event Sourced Services 1) Receive and verify Command (“ApprovePayment”)
123. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”)
124. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log
125. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
126. Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
127. SAD Path - recovering from failure Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
128. SAD Path - recovering from failure Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log
129. SAD Path - recovering from failure Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log 2) Update internal component state
130. SAD Path - recovering from failure Happy Path Event Sourced Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log 2) Update internal component state Memory Image
131. Event Sourcing
132. Event Sourcing ✴ One single Source of Truth with All history
133. Event Sourcing ✴ One single Source of Truth with All history ✴ Allows for Memory Image - durable in-memory state
134. Event Sourcing ✴ One single Source of Truth with All history ✴ Allows for Memory Image - durable in-memory state ✴ Avoids the Object-relational mismatch
135. Event Sourcing ✴ One single Source of Truth with All history ✴ Allows for Memory Image - durable in-memory state ✴ Avoids the Object-relational mismatch ✴ Allows others to Subscribe to state changes
136. Event Sourcing ✴ One single Source of Truth with All history ✴ Allows for Memory Image - durable in-memory state ✴ Avoids the Object-relational mismatch ✴ Allows others to Subscribe to state changes ✴ Mechanical sympathy (Single Writer Principle etc.)
137. ✴ Unfamiliar model ✴ Versioning of events ✴ Deletion of events (legal or moral reasons) Disadvantages Of Using Event Sourcing
138. Events Allow Us To Manage Time
139. “Modeling events forces you to have a temporal focus on what’s going on in the system. Time becomes a crucial factor of the system.” - Greg Young A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
140. But, What is Time Really?
141. Causality How We Make Sense Of The World
142. Time Is the Succession Of Causally Related Events
143. ✴ Event is a snapshot in time ✴ Event ID is an index for time ✴ Event Log is our full history The database of Our past The path to Our present Event Sourcing Allows Us To Model Time
144. Event Sourcing Allows For Time Travel
145. Event Sourcing Allows For Time Travel
146. Event Sourcing Allows For Time Travel ✴Replay the log for historic debugging ✴Replay the log for auditing & traceability ✴Replay the log on failure ✴Replay the log for replication
147. “The future is a function of the past.” - A. P. Robertson
148. “The (local) present is a merge function of multiple concurrent pasts.” - me
149. val newLocalPresent = observedPasts. foldLeft(oldLocalPresent) { _ merge _ }
150. We need to explicitly model the local present as facts derived from the merging of multiple concurrent pasts
151. We Can Even Fork the Past ...Or Join Two Distinct Pasts
152. Let Make Our Demo Event Sourced
153. Untangle Your Read and Write Models With CQRS
154. CQRS
155. CQRS
156. CQRS Write Side Model Commands
157. CQRS Write Side Model Write to Event Log Commands
158. CQRS Write Side Model Events Write to Event Log Commands
159. CQRS Write Side Model Events Events Write to Event Log Commands
160. CQRS Read Side Model Write Side Model Events Events Read Data Store Write to Event Log Commands
161. CQRS Read Side Model Write Side Model Events Queries Events Read Data Store Write to Event Log Commands
162. CQRS Read Side Model Write Side Model Events Queries Events Read Data Store Write to Event Log Eventual Consistency Commands
163. Advantages Of Using CQRS
164. ✴ Temporal Decoupling Advantages Of Using CQRS
165. ✴ Temporal Decoupling ✴ Increased Resilience Advantages Of Using CQRS
166. ✴ Temporal Decoupling ✴ Increased Resilience ✴ Increased Scalability Advantages Of Using CQRS
167. ✴ Temporal Decoupling ✴ Increased Resilience ✴ Increased Scalability ✴ Allows for Multiple Event subscribers Advantages Of Using CQRS
168. ✴ Temporal Decoupling ✴ Increased Resilience ✴ Increased Scalability ✴ Allows for Multiple Event subscribers ✴ Allows for Polyglot Persistence Advantages Of Using CQRS
169. ✴ More infrastructure to manage ✴ 2 data models (or more) to maintain ✴ Eventual Consistency Disadvantages Of Using CQRS
170. Key Takeaways
171. Key Takeaways Event driven design helps you to: ✴ Design autonomous services ✴ Move Faster towards a Reactive architecture ✴ reduce risk when modernizing applications ✴ Balance Certainty and Uncertainty
172. Key Takeaways Event driven design helps you to: ✴ Design autonomous services ✴ Move Faster towards a Reactive architecture ✴ reduce risk when modernizing applications ✴ Balance Certainty and Uncertainty Event Logging allows you to: ✴ take control of your system’s history ✴ time-travel ✴ Balance Strong and eventual consistency
173. Learn More Download my latest book for free at: bit.ly/reactive-microsystems