How Events Are Reshaping Modern Systems

How Events Are Reshaping Modern Systems

Event-driven architecture and design have been getting a lot of attention in recent years. It’s an old concept that has been around for decades, so why this sudden peak of interest? 

In this talk, we will explore the nature of events, what it means to be event-driven, and how we can unleash the power of events. The goal is to arm you with a solid theoretical understanding of how to design an event-driven system, what tools and techniques you can use to reap the most benefit from its design, and perhaps most importantly, what to avoid.

We'll discuss how an event-driven design can help: 
- drive autonomy
- reduce risk
- increase certainty
- increase loose coupling
- increase scalability
- increase resilience
- increase traceability

Skeptics should definitely attend.

E0b5787d1a1935a2800e0bbffc81c196?s=128

Jonas Bonér

October 20, 2017
Tweet

Transcript

  1. How Events Are Reshaping Modern Systems Jonas Bonér @jboner

  2. Why Events?

  3. 1. Events drive autonomy 2. Events help reduce risk 3.

    Events help you move faster 4. Events Increase Loose coupling 5. Events increase stability 6. Events increase scalability 7. Events increase resilience 8. Events increase traceability 9. Events allow for time-travel Why Should you care?
  4. Why Now? 1. Cloud and multicore architectures 2. Microservices and

    distributed systems 3. Data-centric applications 4. “We want more, of everything, and we want it now.” - Your Customers
  5. Do Not Settle For Microliths

  6. Do Not Settle For Microliths

  7. “Event-Driven Architecture (EDA) is a design paradigm in which a

    software component executes in response to receiving one or more event notifications.” - Gartner Event Driven Architecture To The Rescue
  8. What is an Event?

  9. Event “Something that happens.” - Merriam Webster

  10. The Nature of Events

  11. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow The Nature of Events
  12. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored The Nature of Events
  13. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) The Nature of Events
  14. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) ✴ Events/Facts Can not be deleted (once accepted) ➡ Might be needed for legal or moral reasons The Nature of Events
  15. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) ✴ Events/Facts Can not be deleted (once accepted) ➡ Might be needed for legal or moral reasons ✴ Events/Facts (new) can invalidate existing Facts The Nature of Events
  16. Event Loop What Makes Event Tick

  17. ✴ Queue + anonymous Callbacks ✴ The Hollywood principle ✴

    At-Most-Once delivery guarantees ✴ Local context only ✴ Sync or AsynC ✴ Limited and quite powerless The Event Loop
  18. ✴ Queue + anonymous Callbacks ✴ The Hollywood principle ✴

    At-Most-Once delivery guarantees ✴ Local context only ✴ Sync or AsynC ✴ Limited and quite powerless The Event Loop * * Too limited as a general model of computation, and programming model for modern distributed systems. But is a useful building block.
  19. The Problem With Event Callbacks

  20. ✴ Ephemeral and anonymous The Problem With Event Callbacks

  21. ✴ Ephemeral and anonymous ✴ No sane failure model Error

    Channel - console.log(‘4/0=err’, err) The Problem With Event Callbacks
  22. ✴ Ephemeral and anonymous ✴ No sane failure model Error

    Channel - console.log(‘4/0=err’, err) ✴ No composition Signature: Input => Side-effect - (Any => Unit) The Problem With Event Callbacks
  23. ✴ Ephemeral and anonymous ✴ No sane failure model Error

    Channel - console.log(‘4/0=err’, err) ✴ No composition Signature: Input => Side-effect - (Any => Unit) ✴ Hard to Debug The Problem With Event Callbacks
  24. ✴ Ephemeral and anonymous ✴ No sane failure model Error

    Channel - console.log(‘4/0=err’, err) ✴ No composition Signature: Input => Side-effect - (Any => Unit) ✴ Hard to Debug ✴ Hard to reason about The Problem With Event Callbacks
  25. ✴ Ephemeral and anonymous ✴ No sane failure model Error

    Channel - console.log(‘4/0=err’, err) ✴ No composition Signature: Input => Side-effect - (Any => Unit) ✴ Hard to Debug ✴ Hard to reason about ✴ Leads to Callback hell The Problem With Event Callbacks
  26. https://techbrunch.gousto.co.uk/2016/04/22/callback-hell/

  27. None
  28. ✴ Patterns Observer, Pub-Sub, Broadcast, … ✴ Futures & PRomises

    ✴ Dataflow Variables ✴ Stream Processing ✴ Event-driven Microservices ✴ FaaS We Need High Level Abstractions
  29. Unified Event Driven Abstractions Powered By Message Driven Fabric Ladder

    of abstraction
  30. Unified Event Driven Abstractions Powered By Message Driven Fabric Network

    Boundary Ladder of abstraction
  31. Unified Event Driven Abstractions Powered By Message Driven Fabric Network

    Boundary Ladder of abstraction
  32. Unified Event Driven Abstractions Powered By Message Driven Fabric Network

    Boundary Ladder of abstraction
  33. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Network Boundary Ladder of abstraction
  34. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Network Boundary Event 1 Ladder of abstraction
  35. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Network Boundary Event 1 Ladder of abstraction
  36. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Network Boundary Event 1 Ladder of abstraction
  37. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Distribution Fabric Network Boundary Message Passing Event 1 Ladder of abstraction
  38. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Ladder of abstraction
  39. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Ladder of abstraction
  40. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Event 1 Ladder of abstraction
  41. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Event 1 Reactive Systems is doing the heavy lifting Ladder of abstraction
  42. Unified Event Driven Abstractions Powered By Message Driven Fabric Unified

    API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…) Local Event Loop Distribution Fabric Local Event Loop Distribution Fabric Network Boundary Message Passing Event 1 Event 1 Reactive Systems is doing the heavy lifting Ladder of abstraction Reactive Programming
  43. 1. REceive and react to facts (or not), that are

    coming its way 2. Publish new facts (as events) to the rest of the world 3. Invert the control flow to minimize coupling and increase autonomy Event Driven Services
  44. Mutable State Needs To Be Contained And Non Observable

  45. Publish Facts To Outside World

  46. Event Stream

  47. Event Stream Use The as the integration fabric

  48. Event Driven Services

  49. Event Driven Services

  50. Event Driven Services Command

  51. Event Driven Services Command

  52. Event Driven Services Command

  53. Event Driven Services Command Event Event Stream

  54. Event Driven Services Command Event Event Event Event Stream

  55. Event Driven Services Command Event Event Event Event Stream

  56. Event Driven Services Command Event Event Event Event Stream

  57. Event Driven Services Command Event Event Event Event Stream

  58. Event Driven Services Command Event Event Event Event Stream

  59. Event Driven Services Eventual Consistency Command Event Event Event Event

    Stream
  60. Event Driven Services Eventual Consistency Command Event Event Event Event

    Stream
  61. Eventual Consistency

  62. Eventual Consistency No One Wants It’s a necessery evil

  63. But Relax It Is How The World Works

  64. Information Has Latency

  65. Information Is Always From the Past

  66. Welcome To The Wild Ocean Of Non Determinism Distributed Systems

  67. “In a system which cannot count on distributed transactions, the

    management of uncertainty must be implemented in the business logic.” - Pat Helland Life Beyond Distributed Transactions - An Apostate’s Opinion, Pat Helland (2007) We Need To Model Uncertainty
  68. Exploit Reality

  69. Exploit Reality We need to In our design

  70. Events Can Lead To Greater Certainty

  71. Events Can Help Us Craft Autonomous Islands Of Determinism

  72. None
  73. What does it mean to be Automonous?

  74. From greek: Auto-nomos ✴ Auto meaning self ✴ Nomos meaning

    Law Autonomy
  75. Promise Theory

  76. Promise Theory Let Lead the way

  77. ✴Impositions diverge into unpredictable outcomes from definite beginnings 㱺 Distributed

    information 㱺 Decreased certainty Convergence vs. Divergence of information ✴Promises converge towards a definite outcome from unpredictable beginnings 㱺 Local information 㱺 Improved certainty
  78. “Autonomy makes information local, leading to greater certainty and stability.”

    “An autonomus component can only promise its own behavior.” - Mark Burgess In Search of Certainty, Thinking in Promises - Mark Burgess
  79. Benefits of Thinking in Promises

  80. ✴ If a service only promises its own behavior Then

    all information needed to ➡ Resolve a conflict ➡ Repair under failure ➡ Is available within the service itself Benefits of Thinking in Promises
  81. ✴ If a service only promises its own behavior Then

    all information needed to ➡ Resolve a conflict ➡ Repair under failure ➡ Is available within the service itself ✴ Promises remove the need for needless 1. Communication 2. Coordination 3. Consensus Benefits of Thinking in Promises
  82. “The first principle of successful scalability is to batter the

    consistency mechanisms down to a minimum.” - James Hamilton As transcribed in: Toward a Cloud Computing Research Agenda. SIGACT News, 40(2):68–80, June 2009.
  83. Events First Design Drives Autonomy

  84. Events First Domain Driven Design

  85. “When you start modeling events, it forces you to think

    about the behavior of the system. As opposed to thinking about the structure of the system.” - Greg Young A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
  86. ✴ Don’t focus on the things The Nouns The Domain

    Objects ✴ Focus on what happens The Verbs The Events
  87. Mine the Facts

  88. None
  89. Event Storming

  90. Event Driven Design

  91. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts

    ➡ Control Transfer Event Driven Design
  92. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts

    ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer
  93. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts

    ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer Commands
  94. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts

    ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer Commands Events
  95. Event Driven Design

  96. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder,

    ShipProduct Event Driven Design
  97. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder,

    ShipProduct ✴Reactions ➡ Represents side-effects Event Driven Design
  98. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder,

    ShipProduct ✴Reactions ➡ Represents side-effects ✴Events ➡ Represents something that has happened ➡ Past-tense: OrderCreated, ProductShipped Event Driven Design
  99. Commands Events vs

  100. Commands Events vs 1. All about intent 1. Intentless

  101. Commands Events vs 1. All about intent 2. Directed 1.

    Intentless 2. Anonymous
  102. Commands Events vs 1. All about intent 2. Directed 3.

    Single addressable destination 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe
  103. Commands Events vs 1. All about intent 2. Directed 3.

    Single addressable destination 4. Models personal communication 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner)
  104. Commands Events vs 1. All about intent 2. Directed 3.

    Single addressable destination 4. Models personal communication 5. Distributed focus 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner) 5. Local focus
  105. Commands Events vs 1. All about intent 2. Directed 3.

    Single addressable destination 4. Models personal communication 5. Distributed focus 6. Command & Control 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner) 5. Local focus 6. Autonomy
  106. Let the Events Define the Bounded Context

  107. Principles For Promise Theory Driven Protocol Design

  108. 1. Convergency Promises should reach a desired state, a stable

    outcome - idempotent behavior Principles For Promise Theory Driven Protocol Design
  109. 1. Convergency Promises should reach a desired state, a stable

    outcome - idempotent behavior 2. Embracing Failure Expect things to go wrong Promises may or may not be kept Principles For Promise Theory Driven Protocol Design
  110. 1. Convergency Promises should reach a desired state, a stable

    outcome - idempotent behavior 2. Embracing Failure Expect things to go wrong Promises may or may not be kept 3. Autonomy You can’t rely on other services Principles For Promise Theory Driven Protocol Design
  111. ✴ Maintains Integrity & consistency ✴ Is our Unit of

    Consistency ✴ Is our Unit of Failure ✴ Is our Unit of Determinism ✴ Is fully Autonomous The Aggregate
  112. Enough Talk Show Me The Code Sample in Java: https://gist.github.com/jboner/4361c710d2e2386be8bddc39edb0528a

    Sample in Scala: https://gist.github.com/jboner/dea4aa819e127d543d684208d74081b5
  113. Enough Talk Show Me The Code Sample in Java: https://gist.github.com/jboner/4361c710d2e2386be8bddc39edb0528a

    Sample in Scala: https://gist.github.com/jboner/dea4aa819e127d543d684208d74081b5
  114. Are we done now? Perhaps. We have come a long

    way. But events can also be used for: ➡ Persistence ➡ Managing time
  115. Event Based Persistence

  116. “Update-in-place strikes systems designers as a cardinal sin: it violates

    traditional accounting practices that have been observed for hundreds of years.” - jim Gray The Transaction Concept, Jim Gray (1981)
  117. Event Logging The Bedrock

  118. Event Sourcing A Cure For the Cardinal Sin

  119. “The truth is the log. The database is a cache

    of a subset of the log.” - Pat Helland Immutability Changes Everything, Pat Helland (2015)
  120. Event Sourced Services

  121. Happy Path Event Sourced Services

  122. Happy Path Event Sourced Services 1) Receive and verify Command

    (“ApprovePayment”)
  123. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”)

    1) Receive and verify Command (“ApprovePayment”)
  124. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”)

    1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log
  125. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”)

    1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
  126. Happy Path Event Sourced Services 5) Run side-effects (approve the

    payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
  127. SAD Path - recovering from failure Happy Path Event Sourced

    Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
  128. SAD Path - recovering from failure Happy Path Event Sourced

    Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log
  129. SAD Path - recovering from failure Happy Path Event Sourced

    Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log 2) Update internal component state
  130. SAD Path - recovering from failure Happy Path Event Sourced

    Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log 2) Update internal component state Memory Image
  131. Event Sourcing

  132. Event Sourcing ✴ One single Source of Truth with All

    history
  133. Event Sourcing ✴ One single Source of Truth with All

    history ✴ Allows for Memory Image - durable in-memory state
  134. Event Sourcing ✴ One single Source of Truth with All

    history ✴ Allows for Memory Image - durable in-memory state ✴ Avoids the Object-relational mismatch
  135. Event Sourcing ✴ One single Source of Truth with All

    history ✴ Allows for Memory Image - durable in-memory state ✴ Avoids the Object-relational mismatch ✴ Allows others to Subscribe to state changes
  136. Event Sourcing ✴ One single Source of Truth with All

    history ✴ Allows for Memory Image - durable in-memory state ✴ Avoids the Object-relational mismatch ✴ Allows others to Subscribe to state changes ✴ Mechanical sympathy (Single Writer Principle etc.)
  137. ✴ Unfamiliar model ✴ Versioning of events ✴ Deletion of

    events (legal or moral reasons) Disadvantages Of Using Event Sourcing
  138. Events Allow Us To Manage Time

  139. “Modeling events forces you to have a temporal focus on

    what’s going on in the system. Time becomes a crucial factor of the system.” - Greg Young A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
  140. But, What is Time Really?

  141. Causality How We Make Sense Of The World

  142. Time Is the Succession Of Causally Related Events

  143. ✴ Event is a snapshot in time ✴ Event ID

    is an index for time ✴ Event Log is our full history The database of Our past The path to Our present Event Sourcing Allows Us To Model Time
  144. Event Sourcing Allows For Time Travel

  145. Event Sourcing Allows For Time Travel

  146. Event Sourcing Allows For Time Travel ✴Replay the log for

    historic debugging ✴Replay the log for auditing & traceability ✴Replay the log on failure ✴Replay the log for replication
  147. “The future is a function of the past.” - A.

    P. Robertson
  148. “The (local) present is a merge function of multiple concurrent

    pasts.” - me
  149. val newLocalPresent = observedPasts. foldLeft(oldLocalPresent) { _ merge _ }

  150. We need to explicitly model the local present as facts

    derived from the merging of multiple concurrent pasts
  151. We Can Even Fork the Past ...Or Join Two Distinct

    Pasts
  152. Let Make Our Demo Event Sourced

  153. Untangle Your Read and Write Models With CQRS

  154. CQRS

  155. CQRS

  156. CQRS Write Side Model Commands

  157. CQRS Write Side Model Write to Event Log Commands

  158. CQRS Write Side Model Events Write to Event Log Commands

  159. CQRS Write Side Model Events Events Write to Event Log

    Commands
  160. CQRS Read Side Model Write Side Model Events Events Read

    Data Store Write to Event Log Commands
  161. CQRS Read Side Model Write Side Model Events Queries Events

    Read Data Store Write to Event Log Commands
  162. CQRS Read Side Model Write Side Model Events Queries Events

    Read Data Store Write to Event Log Eventual Consistency Commands
  163. Advantages Of Using CQRS

  164. ✴ Temporal Decoupling Advantages Of Using CQRS

  165. ✴ Temporal Decoupling ✴ Increased Resilience Advantages Of Using CQRS

  166. ✴ Temporal Decoupling ✴ Increased Resilience ✴ Increased Scalability Advantages

    Of Using CQRS
  167. ✴ Temporal Decoupling ✴ Increased Resilience ✴ Increased Scalability ✴

    Allows for Multiple Event subscribers Advantages Of Using CQRS
  168. ✴ Temporal Decoupling ✴ Increased Resilience ✴ Increased Scalability ✴

    Allows for Multiple Event subscribers ✴ Allows for Polyglot Persistence Advantages Of Using CQRS
  169. ✴ More infrastructure to manage ✴ 2 data models (or

    more) to maintain ✴ Eventual Consistency Disadvantages Of Using CQRS
  170. Key Takeaways

  171. Key Takeaways Event driven design helps you to: ✴ Design

    autonomous services ✴ Move Faster towards a Reactive architecture ✴ reduce risk when modernizing applications ✴ Balance Certainty and Uncertainty
  172. Key Takeaways Event driven design helps you to: ✴ Design

    autonomous services ✴ Move Faster towards a Reactive architecture ✴ reduce risk when modernizing applications ✴ Balance Certainty and Uncertainty Event Logging allows you to: ✴ take control of your system’s history ✴ time-travel ✴ Balance Strong and eventual consistency
  173. Learn More Download my latest book for free at: bit.ly/reactive-microsystems