Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Designing Events-first Microservices

Designing Events-first Microservices

In this talk, we will explore the nature of events, what it means to be event-driven, and how we can unleash the power of events and commands by applying an events-first domain-driven design to microservices-based architectures.

We will start by developing a solid theoretical understanding of how to design systems of event-driven microservices. Then we will discuss the practical tools and techniques you can use to reap the most benefit from that design, as well as, most importantly, what to avoid along the way.

We’ll discuss how an events-first design approach to building microservices can improve the following characteristics over competing techniques:
- increase certainty
- increase resilience
- increase scalability
- increase traceability
- increase loose coupling
- reduce risk

Skeptics should definitely attend.

E0b5787d1a1935a2800e0bbffc81c196?s=128

Jonas Bonér

June 29, 2018
Tweet

Transcript

  1. Designing Events First Microservices Jonas Bonér @jboner

  2. 1. Events drive autonomy 2. Events help reduce risk 3.

    Events help you move faster 4. Events Increase Loose coupling 5. Events increase stability 6. Events increase scalability 7. Events increase resilience 8. Events increase traceability 9. Events allow for time-travel Why Should you care about Events?
  3. Why Now? 1. Cloud and multicore architectures 2. Microservices and

    distributed systems 3. Data-centric applications - Data in motion 4. “We want more, of everything, and we want it now.” -Your Customers
  4. So, you want to do microservices?

  5. Make sure you don’t end up with Microliths

  6. Make sure you don’t end up with Microliths We can

    do better than this
  7. Events First Domain Driven Design

  8. “When you start modeling events, it forces you to think

    about the behaviour of the system. As opposed to thinking about the structure of the system.” - Greg Young A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
  9. ✴ Don’t focus on the things The Nouns The Domain

    Objects
  10. ✴ Don’t focus on the things The Nouns The Domain

    Objects ✴ Focus on what happens The Verbs The Events
  11. What is an Event?

  12. The Nature of Events

  13. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow The Nature of Events
  14. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored The Nature of Events
  15. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) The Nature of Events
  16. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) ✴ Events/Facts Can not be deleted (once accepted) ➡ Might be needed for legal or moral reasons The Nature of Events
  17. ✴ Events represent Facts of information ➡ Facts are immutable

    ➡ Facts Accrue - Knowledge can only grow ✴ Events/Facts can be disregarded/Ignored ✴ Events/Facts Can not be retracted (once accepted) ✴ Events/Facts Can not be deleted (once accepted) ➡ Might be needed for legal or moral reasons ✴ Events/Facts (new) can invalidate existing Facts The Nature of Events
  18. Mine the Facts

  19. None
  20. Event Storming

  21. Event Driven Design

  22. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts

    ➡ Control Transfer Event Driven Design
  23. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts

    ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer
  24. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts

    ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer Commands
  25. ✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts

    ➡ Control Transfer Event Driven Design ✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer Commands Events
  26. Event Driven Design

  27. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder,

    ShipProduct Event Driven Design
  28. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder,

    ShipProduct ✴Reactions ➡ Represents side-effects Event Driven Design
  29. ✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder,

    ShipProduct ✴Reactions ➡ Represents side-effects ✴Events ➡ Represents something that has happened ➡ Past-tense: OrderCreated, ProductShipped Event Driven Design
  30. Commands Events vs

  31. Commands Events vs 1. All about intent 1. Intentless

  32. Commands Events vs 1. All about intent 2. Directed 1.

    Intentless 2. Anonymous
  33. Commands Events vs 1. All about intent 2. Directed 3.

    Single addressable destination 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe
  34. Commands Events vs 1. All about intent 2. Directed 3.

    Single addressable destination 4. Models personal communication 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner)
  35. Commands Events vs 1. All about intent 2. Directed 3.

    Single addressable destination 4. Models personal communication 5. Distributed focus 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner) 5. Local focus
  36. Commands Events vs 1. All about intent 2. Directed 3.

    Single addressable destination 4. Models personal communication 5. Distributed focus 6. Command & Control 1. Intentless 2. Anonymous 3. Just happens - for others (0-N) to observe 4. Models broadcast (speakers corner) 5. Local focus 6. Autonomy
  37. Let the Events Define the Bounded Context

  38. Event Driven Services

  39. 1. REceive & react (or not) 
 to facts* that

    are coming its way Event Driven Services
  40. 1. REceive & react (or not) 
 to facts* that

    are coming its way 2. Publish new facts* asynchronously 
 to the rest of the world Event Driven Services
  41. 1. REceive & react (or not) 
 to facts* that

    are coming its way 2. Publish new facts* asynchronously 
 to the rest of the world 3. Invert the control flow
 to minimize coupling and increase autonomy Event Driven Services
  42. 1. REceive & react (or not) 
 to facts* that

    are coming its way 2. Publish new facts* asynchronously 
 to the rest of the world 3. Invert the control flow
 to minimize coupling and increase autonomy Event Driven Services *Facts == Immutable Events
  43. Mutable State Is Fine But Needs To Be Contained And

    Non Observable
  44. Publish Facts To Outside World

  45. ✴ Maintains Integrity & consistency ✴ Is our Unit of

    Consistency ✴ Is our Unit of Failure ✴ Is our Unit of Determinism ✴ Is fully Autonomous The Aggregate
  46. Event Driven Services

  47. Event Driven Services

  48. Event Driven Services Command

  49. Event Driven Services Command

  50. Event Driven Services Command

  51. Event Driven Services Command Event Event Stream

  52. Event Driven Services Command Event Event Event Event Stream

  53. Event Driven Services Command Event Event Event Event Stream

  54. Event Driven Services Command Event Event Event Event Stream

  55. Event Driven Services Command Event Event Event Event Stream

  56. Event Driven Services Command Event Event Event Event Stream

  57. Event Driven Services Eventual Consistency Command Event Event Event Event

    Stream
  58. Event Stream Use The

  59. Event Stream Use The as the communication fabric

  60. Event Stream Use The

  61. Event Stream Use The as the integration fabric

  62. Event Stream Use The

  63. Event Stream Use The as the replication fabric

  64. Event Stream Use The

  65. Event Stream Use The as the consensus fabric

  66. Event Stream Use The

  67. Event Stream Use The as the Persistence fabric

  68. The Problem With CRUD Services

  69. ✴ CRUD is fine for totally isolated data The Problem

    With CRUD Services
  70. ✴ CRUD is fine for totally isolated data ✴ But,

    cross CRUD services consistency The Problem With CRUD Services
  71. ✴ CRUD is fine for totally isolated data ✴ But,

    cross CRUD services consistency ➡ Is hard ⇨ No Joins The Problem With CRUD Services
  72. ✴ CRUD is fine for totally isolated data ✴ But,

    cross CRUD services consistency ➡ Is hard ⇨ No Joins ➡ Has ad-hoc & weak guarantees The Problem With CRUD Services
  73. ✴ CRUD is fine for totally isolated data ✴ But,

    cross CRUD services consistency ➡ Is hard ⇨ No Joins ➡ Has ad-hoc & weak guarantees The Problem With CRUD Services
  74. “In general, application developers simply do not implement large scalable

    applications assuming distributed transactions” - Pat Helland Life Beyond Distributed Transactions - Pat Helland
  75. “Two-phase commit is the anti-availability protocol.” - Pat Helland Standing

    on Distributed Shoulders of Giants - Pat Helland
  76. STRONG Consistency Is the wrong default In distributed systems

  77. STRONG Consistency Is the wrong default In distributed systems What

    can we do?
  78. Eventual Consistency We have to rely on

  79. Eventual Consistency We have to rely on But relax—it’s how

    the world works
  80. Embrace Reality We Need to

  81. Embrace Reality We Need to Not Fight it

  82. Information Has Latency

  83. Information Is Always From the Past

  84. Welcome To The Wild Ocean Of Non Determinism Distributed Systems

  85. “In a system which cannot count on distributed transactions, the

    management of uncertainty must be implemented in the business logic.” - Pat Helland Life Beyond Distributed Transactions, Pat Helland (2007) We Need To Model Uncertainty
  86. Events Can Lead To Greater Certainty

  87. “An autonomus component can only promise its own behavior.” “Autonomy

    makes information local, leading to greater certainty and stability.” - Mark Burgess In Search of Certainty, Thinking in Promises - Mark Burgess
  88. Events Can Help Us Craft Autonomous Islands Of Determinism

  89. Data on the inside vs Data on the outside -

    Pat Helland
  90. Data on the inside vs Data on the outside -

    Pat Helland Inside Data Our current present ⇨ state
  91. Data on the inside vs Data on the outside -

    Pat Helland Inside Data Our current present ⇨ state Outside Data Blast from the past ⇨ Events/facts
  92. Data on the inside vs Data on the outside -

    Pat Helland Inside Data Our current present ⇨ state Outside Data Blast from the past ⇨ Events/facts Between Services Hope for the future ⇨ commands
  93. A system of microservices is a never ending stream towards

    convergence
  94. There Is No Now A system of microservices is a

    never ending stream towards convergence
  95. Resilience is by Design Photo courtesy of FEMA/Joselyne Augustino

  96. Events Can Help Us Manage Failure Instead Of Trying To

    Avoid It
  97. Requirements for a Sane Failure Model 1. Contained—Avoid cascading failures

    2. Reified—as Events 3. Signalled—Asynchronously 4. Observed—by 1-N 5. Managed—Outside failed Context Failures need to be
  98. Event Based Persistence

  99. You can use CRUD Together with Event Streams To get

    an internally consistent Materialized View
  100. Service B Service A You can use CRUD Together with

    Event Streams To get an internally consistent Materialized View
  101. Service B Service A CRUD You can use CRUD Together

    with Event Streams To get an internally consistent Materialized View CRUD
  102. Service B Service A TABLE A CRUD TABLE B You

    can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD
  103. Service B Service A TABLE A CRUD TABLE B You

    can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD
  104. Service B Service A TABLE A CRUD TABLE B You

    can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
  105. Service B Service A TABLE A CRUD TABLE B You

    can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
  106. Service C Service B Service A TABLE A CRUD TABLE

    B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
  107. Service C Service B Service A TABLE A CRUD TABLE

    B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
  108. Service C Service B Service A TABLE A CRUD TABLE

    B JOINS Table A Table B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Atomic Update & event
  109. Service C Service B Service A TABLE A CRUD TABLE

    B JOINS Table A Table B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Read Only Atomic Update & event
  110. Service C Service B Service A Eventual Consistency TABLE A

    CRUD TABLE B JOINS Table A Table B You can use CRUD Together with Event Streams To get an internally consistent Materialized View CRUD Read Only Atomic Update & event
  111. “Update-in-place strikes systems designers as a cardinal sin: it violates

    traditional accounting practices that have been observed for hundreds of years.” - jim Gray The Transaction Concept, Jim Gray (1981)
  112. “The truth is the log. The database is a cache

    of a subset of the log.” - Pat Helland Immutability Changes Everything, Pat Helland (2015)
  113. Event Logging The Bedrock

  114. Event Sourcing A Cure For the Cardinal Sin

  115. Event Sourced Services

  116. Happy Path Event Sourced Services

  117. Happy Path Event Sourced Services 1) Receive and verify Command

    (“ApprovePayment”)
  118. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”)

    1) Receive and verify Command (“ApprovePayment”)
  119. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”)

    1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log
  120. Happy Path Event Sourced Services 2) Create new Event (“PaymentApproved”)

    1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
  121. Happy Path Event Sourced Services 5) Run side-effects (approve the

    payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
  122. SAD Path - recover from failure Happy Path Event Sourced

    Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state
  123. SAD Path - recover from failure Happy Path Event Sourced

    Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log
  124. SAD Path - recover from failure Happy Path Event Sourced

    Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log 2) Update internal component state
  125. SAD Path - recover from failure Happy Path Event Sourced

    Services 5) Run side-effects (approve the payment) 2) Create new Event (“PaymentApproved”) 1) Receive and verify Command (“ApprovePayment”) 3) Append Event to Event Log 4) Update internal component state 1) Rehydrate Events from Event Log 2) Update internal component state Memory Image
  126. Event Sourcing

  127. Event Sourcing ✴ One single Source of Truth with All

    history
  128. Event Sourcing ✴ One single Source of Truth with All

    history ✴ Allows for Memory Image (Durable In-Memory State)
  129. Event Sourcing ✴ One single Source of Truth with All

    history ✴ Allows for Memory Image (Durable In-Memory State) ✴ Avoids the Object-relational mismatch
  130. Event Sourcing ✴ One single Source of Truth with All

    history ✴ Allows for Memory Image (Durable In-Memory State) ✴ Avoids the Object-relational mismatch ✴ Allows others to Subscribe to state changes
  131. Event Sourcing ✴ One single Source of Truth with All

    history ✴ Allows for Memory Image (Durable In-Memory State) ✴ Avoids the Object-relational mismatch ✴ Allows others to Subscribe to state changes ✴ Has good Mechanical sympathy
 (Single Writer Principle etc.)
  132. ✴ Unfamiliar model ✴ Versioning of events ✴ Deletion of

    events (legal or moral reasons) Disadvantages Of Using Event Sourcing
  133. Untangle Your Read and Write Models With CQRS

  134. CQRS

  135. CQRS

  136. CQRS Write Side Model Commands

  137. CQRS Write Side Model Write to Event Log Commands

  138. CQRS Write Side Model Events Write to Event Log Commands

  139. CQRS Write Side Model Events Events Write to Event Log

    Commands
  140. CQRS Read Side Model Write Side Model Events Events Read

    Data Store Write to Event Log Commands
  141. CQRS Read Side Model Write Side Model Events Queries Events

    Read Data Store Write to Event Log Commands
  142. CQRS Read Side Model Write Side Model Events Queries Events

    Read Data Store Write to Event Log Eventual Consistency Commands
  143. ✴ Temporal Decoupling ✴ Increased Resilience ✴ Increased Scalability ✴

    Allows for Multiple Event subscribers ✴ Allows for Polyglot Persistence Advantages Of Using CQRS
  144. ✴ More infrastructure to manage ✴ 2 data models (or

    more) to maintain ✴ Eventual Consistency Disadvantages Of Using CQRS
  145. Events Allow Us To Manage Time

  146. “Modelling events forces you to have a 
 temporal focus

    on what’s going on in the system. Time becomes a crucial factor of the system.” - Greg Young A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
  147. ✴ Event is a snapshot in time ✴ Event ID

    is an index for time ✴ Event Log is our full history The database of Our past The path to Our present Event Sourcing Allows Us To Model Time
  148. Event Sourcing Allows For Time Travel

  149. Event Sourcing Allows For Time Travel

  150. Event Sourcing Allows For Time Travel ✴Replay the log for

    historic debugging ✴Replay the log for auditing & traceability ✴Replay the log on failure ✴Replay the log for replication
  151. We Can Even Fork the Past ...Or Join Two Distinct

    Pasts
  152. Key Takeaways

  153. Key Takeaways Events-First design helps you to: ✴ reduce risk

    when modernizing applications ✴ Move Faster towards a Resilient and Scalable architecture ✴ Design autonomous services ✴ Balance Certainty and Uncertainty
  154. Key Takeaways Events-First design helps you to: ✴ reduce risk

    when modernizing applications ✴ Move Faster towards a Resilient and Scalable architecture ✴ Design autonomous services ✴ Balance Certainty and Uncertainty Event Logging allows you to: ✴ AVOID CRUD and ORM ✴ take control of your system’s history ✴ time-travel ✴ Balance Strong and eventual consistency
  155. http://akka.io

  156. Learn More Download my latest book for free at: bit.ly/reactive-microsystems