Designing Events-first Microservices

In this talk, we will explore the nature of events, what it means to be event-driven, and how we can unleash the power of events and commands by applying an events-first domain-driven design to microservices-based architectures.

We will start by developing a solid theoretical understanding of how to design systems of event-driven microservices. Then we will discuss the practical tools and techniques you can use to reap the most benefit from that design, as well as, most importantly, what to avoid along the way.

We’ll discuss how an events-first design approach to building microservices can improve the following characteristics over competing techniques:
- increase certainty
- increase resilience
- increase scalability
- increase traceability
- increase loose coupling
- reduce risk

Skeptics should definitely attend.

Transcript

1. Designing
   Events First
   Microservices
   Jonas Bonér
   @jboner
   

   View Slide

2. 1. Events drive autonomy
   2. Events help reduce risk
   3. Events help you move faster
   4. Events Increase Loose coupling
   5. Events increase stability
   6. Events increase scalability
   7. Events increase resilience
   8. Events increase traceability
   9. Events allow for time-travel
   Why Should you care about Events?
   

   View Slide

3. Why Now?
   1. Cloud and multicore architectures
   2. Microservices and distributed systems
   3. Data-centric applications - Data in motion
   4. “We want more, of everything, and we
   want it now.” -Your Customers
   

   View Slide

4. So, you want to do
   microservices?
   

   View Slide

5. Make sure you don’t end up with
   Microliths
   

   View Slide

6. Make sure you don’t end up with
   Microliths
   We can do better than this
   

   View Slide

7. Events First
   Domain Driven
   Design
   

   View Slide

8. “When you start modeling events, it
   forces you to think about the behaviour
   of the system. As opposed to thinking
   about the structure of the system.”
   - Greg Young
   A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
   

   View Slide

9. ✴ Don’t focus on the things
   The Nouns
   The Domain Objects
   

   View Slide

10. ✴ Don’t focus on the things
    The Nouns
    The Domain Objects
    ✴ Focus on what happens
    The Verbs
    The Events
    

    View Slide

11. What is an
    Event?
    

    View Slide

12. The Nature of Events
    

    View Slide

13. ✴ Events represent Facts of information
    ➡ Facts are immutable
    ➡ Facts Accrue - Knowledge can only grow
    The Nature of Events
    

    View Slide

14. ✴ Events represent Facts of information
    ➡ Facts are immutable
    ➡ Facts Accrue - Knowledge can only grow
    ✴ Events/Facts can be disregarded/Ignored
    The Nature of Events
    

    View Slide

15. ✴ Events represent Facts of information
    ➡ Facts are immutable
    ➡ Facts Accrue - Knowledge can only grow
    ✴ Events/Facts can be disregarded/Ignored
    ✴ Events/Facts Can not be retracted (once accepted)
    The Nature of Events
    

    View Slide

16. ✴ Events represent Facts of information
    ➡ Facts are immutable
    ➡ Facts Accrue - Knowledge can only grow
    ✴ Events/Facts can be disregarded/Ignored
    ✴ Events/Facts Can not be retracted (once accepted)
    ✴ Events/Facts Can not be deleted (once accepted)
    ➡ Might be needed for legal or moral reasons
    The Nature of Events
    

    View Slide

17. ✴ Events represent Facts of information
    ➡ Facts are immutable
    ➡ Facts Accrue - Knowledge can only grow
    ✴ Events/Facts can be disregarded/Ignored
    ✴ Events/Facts Can not be retracted (once accepted)
    ✴ Events/Facts Can not be deleted (once accepted)
    ➡ Might be needed for legal or moral reasons
    ✴ Events/Facts (new) can invalidate existing Facts
    The Nature of Events
    

    View Slide

18. Mine the
    Facts
    

    View Slide

19. View Slide

20. Event Storming
    

    View Slide

21. Event Driven Design
    

    View Slide

22. ✴ IntentS
    ➡ Communication
    ➡ Conversations
    ➡ Expectations
    ➡ Contracts
    ➡ Control Transfer
    Event Driven Design
    

    View Slide

23. ✴ IntentS
    ➡ Communication
    ➡ Conversations
    ➡ Expectations
    ➡ Contracts
    ➡ Control Transfer
    Event Driven Design
    ✴ Facts
    ➡ State
    ➡ History
    ➡ Causality
    ➡ Notifications
    ➡ State Transfer
    

    View Slide

24. ✴ IntentS
    ➡ Communication
    ➡ Conversations
    ➡ Expectations
    ➡ Contracts
    ➡ Control Transfer
    Event Driven Design
    ✴ Facts
    ➡ State
    ➡ History
    ➡ Causality
    ➡ Notifications
    ➡ State Transfer
    Commands
    

    View Slide

25. ✴ IntentS
    ➡ Communication
    ➡ Conversations
    ➡ Expectations
    ➡ Contracts
    ➡ Control Transfer
    Event Driven Design
    ✴ Facts
    ➡ State
    ➡ History
    ➡ Causality
    ➡ Notifications
    ➡ State Transfer
    Commands Events
    

    View Slide

26. Event Driven Design
    

    View Slide

27. ✴Commands
    ➡ Object form of method/Action request
    ➡ Imperative: CreateOrder, ShipProduct
    Event Driven Design
    

    View Slide

28. ✴Commands
    ➡ Object form of method/Action request
    ➡ Imperative: CreateOrder, ShipProduct
    ✴Reactions
    ➡ Represents side-effects
    Event Driven Design
    

    View Slide

29. ✴Commands
    ➡ Object form of method/Action request
    ➡ Imperative: CreateOrder, ShipProduct
    ✴Reactions
    ➡ Represents side-effects
    ✴Events
    ➡ Represents something that has happened
    ➡ Past-tense: OrderCreated, ProductShipped
    Event Driven Design
    

    View Slide

30. Commands Events
    vs
    

    View Slide

31. Commands Events
    vs
    1. All about intent 1. Intentless
    

    View Slide

32. Commands Events
    vs
    1. All about intent
    2. Directed
    1. Intentless
    2. Anonymous
    

    View Slide

33. Commands Events
    vs
    1. All about intent
    2. Directed
    3. Single addressable
    destination
    1. Intentless
    2. Anonymous
    3. Just happens - for
    others (0-N) to observe
    

    View Slide

34. Commands Events
    vs
    1. All about intent
    2. Directed
    3. Single addressable
    destination
    4. Models personal
    communication
    1. Intentless
    2. Anonymous
    3. Just happens - for
    others (0-N) to observe
    4. Models broadcast
    (speakers corner)
    

    View Slide

35. Commands Events
    vs
    1. All about intent
    2. Directed
    3. Single addressable
    destination
    4. Models personal
    communication
    5. Distributed focus
    1. Intentless
    2. Anonymous
    3. Just happens - for
    others (0-N) to observe
    4. Models broadcast
    (speakers corner)
    5. Local focus
    

    View Slide

36. Commands Events
    vs
    1. All about intent
    2. Directed
    3. Single addressable
    destination
    4. Models personal
    communication
    5. Distributed focus
    6. Command & Control
    1. Intentless
    2. Anonymous
    3. Just happens - for
    others (0-N) to observe
    4. Models broadcast
    (speakers corner)
    5. Local focus
    6. Autonomy
    

    View Slide

37. Let the Events Define the
    Bounded Context
    

    View Slide

38. Event Driven
    Services
    

    View Slide

39. 1. REceive & react (or not) 

    to facts* that are coming its way
    Event Driven
    Services
    

    View Slide

40. 1. REceive & react (or not) 

    to facts* that are coming its way
    2. Publish new facts* asynchronously 

    to the rest of the world
    Event Driven
    Services
    

    View Slide

41. 1. REceive & react (or not) 

    to facts* that are coming its way
    2. Publish new facts* asynchronously 

    to the rest of the world
    3. Invert the control flow

    to minimize coupling and increase autonomy
    Event Driven
    Services
    

    View Slide

42. 1. REceive & react (or not) 

    to facts* that are coming its way
    2. Publish new facts* asynchronously 

    to the rest of the world
    3. Invert the control flow

    to minimize coupling and increase autonomy
    Event Driven
    Services
    *Facts == Immutable Events
    

    View Slide

43. Mutable State Is Fine
    But Needs To Be
    Contained
    And Non Observable
    

    View Slide

44. Publish Facts
    To Outside World
    

    View Slide

45. ✴ Maintains Integrity & consistency
    ✴ Is our Unit of Consistency
    ✴ Is our Unit of Failure
    ✴ Is our Unit of Determinism
    ✴ Is fully Autonomous
    The Aggregate
    

    View Slide

46. Event Driven
    Services
    

    View Slide

47. Event Driven
    Services
    

    View Slide

48. Event Driven
    Services
    Command
    

    View Slide

49. Event Driven
    Services
    Command
    

    View Slide

50. Event Driven
    Services
    Command
    

    View Slide

51. Event Driven
    Services
    Command
    Event
    Event Stream
    

    View Slide

52. Event Driven
    Services
    Command
    Event
    Event
    Event Event Stream
    

    View Slide

53. Event Driven
    Services
    Command
    Event
    Event
    Event Event Stream
    

    View Slide

54. Event Driven
    Services
    Command
    Event
    Event
    Event Event Stream
    

    View Slide

55. Event Driven
    Services
    Command
    Event
    Event
    Event Event Stream
    

    View Slide

56. Event Driven
    Services
    Command
    Event
    Event
    Event Event Stream
    

    View Slide

57. Event Driven
    Services
    Eventual
    Consistency
    Command
    Event
    Event
    Event Event Stream
    

    View Slide

58. Event
    Stream
    Use The
    

    View Slide

59. Event
    Stream
    Use The
    as the communication fabric
    

    View Slide

60. Event
    Stream
    Use The
    

    View Slide

61. Event
    Stream
    Use The
    as the integration fabric
    

    View Slide

62. Event
    Stream
    Use The
    

    View Slide

63. Event
    Stream
    Use The
    as the replication fabric
    

    View Slide

64. Event
    Stream
    Use The
    

    View Slide

65. Event
    Stream
    Use The
    as the consensus fabric
    

    View Slide

66. Event
    Stream
    Use The
    

    View Slide

67. Event
    Stream
    Use The
    as the Persistence fabric
    

    View Slide

68. The Problem With
    CRUD Services
    

    View Slide

69. ✴ CRUD is fine for totally isolated data
    The Problem With
    CRUD Services
    

    View Slide

70. ✴ CRUD is fine for totally isolated data
    ✴ But, cross CRUD services consistency
    The Problem With
    CRUD Services
    

    View Slide

71. ✴ CRUD is fine for totally isolated data
    ✴ But, cross CRUD services consistency
    ➡ Is hard ⇨ No Joins
    The Problem With
    CRUD Services
    

    View Slide

72. ✴ CRUD is fine for totally isolated data
    ✴ But, cross CRUD services consistency
    ➡ Is hard ⇨ No Joins
    ➡ Has ad-hoc & weak guarantees
    The Problem With
    CRUD Services
    

    View Slide

73. ✴ CRUD is fine for totally isolated data
    ✴ But, cross CRUD services consistency
    ➡ Is hard ⇨ No Joins
    ➡ Has ad-hoc & weak guarantees
    The Problem With
    CRUD Services
    

    View Slide

74. “In general, application developers simply
    do not implement large scalable applications
    assuming distributed transactions”
    - Pat Helland
    Life Beyond Distributed Transactions - Pat Helland
    

    View Slide

75. “Two-phase commit is the
    anti-availability protocol.”
    - Pat Helland
    Standing on Distributed Shoulders of Giants - Pat Helland
    

    View Slide

76. STRONG
    Consistency
    Is the wrong default
    In distributed systems
    

    View Slide

77. STRONG
    Consistency
    Is the wrong default
    In distributed systems
    What can we do?
    

    View Slide

78. Eventual
    Consistency
    We have to rely on
    

    View Slide

79. Eventual
    Consistency
    We have to rely on
    But relax—it’s how the world works
    

    View Slide

80. Embrace
    Reality
    We Need to
    

    View Slide

81. Embrace
    Reality
    We Need to
    Not Fight it
    

    View Slide

82. Information
    Has Latency
    

    View Slide

83. Information Is Always
    From the Past
    

    View Slide

84. Welcome To The Wild Ocean Of
    Non Determinism
    Distributed Systems
    

    View Slide

85. “In a system which cannot count on
    distributed transactions, the management
    of uncertainty must be implemented in the
    business logic.”
    - Pat Helland
    Life Beyond Distributed Transactions, Pat Helland (2007)
    We Need To Model
    Uncertainty
    

    View Slide

86. Events Can Lead To Greater
    Certainty
    

    View Slide

87. “An autonomus component can only
    promise its own behavior.”
    “Autonomy makes information local,
    leading to greater certainty and stability.”
    - Mark Burgess
    In Search of Certainty, Thinking in Promises - Mark Burgess
    

    View Slide

88. Events Can Help Us Craft
    Autonomous Islands
    Of Determinism
    

    View Slide

89. Data on the inside vs Data on the outside - Pat Helland
    

    View Slide

90. Data on the inside vs Data on the outside - Pat Helland
    Inside Data
    Our current present ⇨ state
    

    View Slide

91. Data on the inside vs Data on the outside - Pat Helland
    Inside Data
    Our current present ⇨ state
    Outside Data
    Blast from the past ⇨ Events/facts
    

    View Slide

92. Data on the inside vs Data on the outside - Pat Helland
    Inside Data
    Our current present ⇨ state
    Outside Data
    Blast from the past ⇨ Events/facts
    Between Services
    Hope for the future ⇨ commands
    

    View Slide

93. A system of microservices is a
    never ending stream towards convergence
    

    View Slide

94. There Is No Now
    A system of microservices is a
    never ending stream towards convergence
    

    View Slide

95. Resilience
    is by
    Design
    Photo courtesy of FEMA/Joselyne Augustino
    

    View Slide

96. Events Can Help Us
    Manage
    Failure
    Instead Of Trying To Avoid It
    

    View Slide

97. Requirements for a
    Sane Failure Model
    1. Contained—Avoid cascading failures
    2. Reified—as Events
    3. Signalled—Asynchronously
    4. Observed—by 1-N
    5. Managed—Outside failed Context
    Failures need to be
    

    View Slide

98. Event Based
    Persistence
    

    View Slide

99. You can use CRUD
    Together with Event Streams
    To get an internally consistent Materialized View
    

    View Slide

100. Service B
     Service A
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     

     View Slide

101. Service B
     Service A
     CRUD
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     

     View Slide

102. Service B
     Service A
     TABLE A
     CRUD
     TABLE B
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     

     View Slide

103. Service B
     Service A
     TABLE A
     CRUD
     TABLE B
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     

     View Slide

104. Service B
     Service A
     TABLE A
     CRUD
     TABLE B
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     Atomic Update & event
     

     View Slide

105. Service B
     Service A
     TABLE A
     CRUD
     TABLE B
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     Atomic Update & event
     

     View Slide

106. Service C
     Service B
     Service A
     TABLE A
     CRUD
     TABLE B
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     Atomic Update & event
     

     View Slide

107. Service C
     Service B
     Service A
     TABLE A
     CRUD
     TABLE B
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     Atomic Update & event
     

     View Slide

108. Service C
     Service B
     Service A
     TABLE A
     CRUD
     TABLE B
     JOINS
     Table A
     Table B
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     Atomic Update & event
     

     View Slide

109. Service C
     Service B
     Service A
     TABLE A
     CRUD
     TABLE B
     JOINS
     Table A
     Table B
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     Read
     Only
     Atomic Update & event
     

     View Slide

110. Service C
     Service B
     Service A
     Eventual
     Consistency
     TABLE A
     CRUD
     TABLE B
     JOINS
     Table A
     Table B
     You can use CRUD
     Together with Event Streams
     To get an internally consistent Materialized View
     CRUD
     Read
     Only
     Atomic Update & event
     

     View Slide

111. “Update-in-place strikes systems
     designers as a cardinal sin: it violates
     traditional accounting practices that have
     been observed for hundreds of years.”
     - jim Gray
     The Transaction Concept, Jim Gray (1981)
     

     View Slide

112. “The truth is the log.
     The database is a cache
     of a subset of the log.”
     - Pat Helland
     Immutability Changes Everything, Pat Helland (2015)
     

     View Slide

113. Event Logging
     The Bedrock
     

     View Slide

114. Event Sourcing
     A Cure For the Cardinal Sin
     

     View Slide

115. Event
     Sourced
     Services
     

     View Slide

116. Happy Path
     Event
     Sourced
     Services
     

     View Slide

117. Happy Path
     Event
     Sourced
     Services
     1) Receive and verify Command
     (“ApprovePayment”)
     

     View Slide

118. Happy Path
     Event
     Sourced
     Services
     2) Create new Event
     (“PaymentApproved”)
     1) Receive and verify Command
     (“ApprovePayment”)
     

     View Slide

119. Happy Path
     Event
     Sourced
     Services
     2) Create new Event
     (“PaymentApproved”)
     1) Receive and verify Command
     (“ApprovePayment”)
     3) Append Event
     to Event Log
     

     View Slide

120. Happy Path
     Event
     Sourced
     Services
     2) Create new Event
     (“PaymentApproved”)
     1) Receive and verify Command
     (“ApprovePayment”)
     3) Append Event
     to Event Log
     4) Update internal
     component state
     

     View Slide

121. Happy Path
     Event
     Sourced
     Services
     5) Run side-effects
     (approve the payment)
     2) Create new Event
     (“PaymentApproved”)
     1) Receive and verify Command
     (“ApprovePayment”)
     3) Append Event
     to Event Log
     4) Update internal
     component state
     

     View Slide

122. SAD Path - recover from failure
     Happy Path
     Event
     Sourced
     Services
     5) Run side-effects
     (approve the payment)
     2) Create new Event
     (“PaymentApproved”)
     1) Receive and verify Command
     (“ApprovePayment”)
     3) Append Event
     to Event Log
     4) Update internal
     component state
     

     View Slide

123. SAD Path - recover from failure
     Happy Path
     Event
     Sourced
     Services
     5) Run side-effects
     (approve the payment)
     2) Create new Event
     (“PaymentApproved”)
     1) Receive and verify Command
     (“ApprovePayment”)
     3) Append Event
     to Event Log
     4) Update internal
     component state
     1) Rehydrate Events
     from Event Log
     

     View Slide

124. SAD Path - recover from failure
     Happy Path
     Event
     Sourced
     Services
     5) Run side-effects
     (approve the payment)
     2) Create new Event
     (“PaymentApproved”)
     1) Receive and verify Command
     (“ApprovePayment”)
     3) Append Event
     to Event Log
     4) Update internal
     component state
     1) Rehydrate Events
     from Event Log
     2) Update internal
     component state
     

     View Slide

125. SAD Path - recover from failure
     Happy Path
     Event
     Sourced
     Services
     5) Run side-effects
     (approve the payment)
     2) Create new Event
     (“PaymentApproved”)
     1) Receive and verify Command
     (“ApprovePayment”)
     3) Append Event
     to Event Log
     4) Update internal
     component state
     1) Rehydrate Events
     from Event Log
     2) Update internal
     component state
     Memory Image
     

     View Slide

126. Event Sourcing
     

     View Slide

127. Event Sourcing
     ✴ One single Source of Truth with All history
     

     View Slide

128. Event Sourcing
     ✴ One single Source of Truth with All history
     ✴ Allows for Memory Image (Durable In-Memory State)
     

     View Slide

129. Event Sourcing
     ✴ One single Source of Truth with All history
     ✴ Allows for Memory Image (Durable In-Memory State)
     ✴ Avoids the Object-relational mismatch
     

     View Slide

130. Event Sourcing
     ✴ One single Source of Truth with All history
     ✴ Allows for Memory Image (Durable In-Memory State)
     ✴ Avoids the Object-relational mismatch
     ✴ Allows others to Subscribe to state changes
     

     View Slide

131. Event Sourcing
     ✴ One single Source of Truth with All history
     ✴ Allows for Memory Image (Durable In-Memory State)
     ✴ Avoids the Object-relational mismatch
     ✴ Allows others to Subscribe to state changes
     ✴ Has good Mechanical sympathy

     (Single Writer Principle etc.)
     

     View Slide

132. ✴ Unfamiliar model
     ✴ Versioning of events
     ✴ Deletion of events (legal or moral reasons)
     Disadvantages
     Of Using Event Sourcing
     

     View Slide

133. Untangle Your
     Read and Write Models
     With CQRS
     

     View Slide

134. CQRS
     

     View Slide

135. CQRS
     

     View Slide

136. CQRS
     Write Side Model
     Commands
     

     View Slide

137. CQRS
     Write Side Model
     Write to Event Log
     Commands
     

     View Slide

138. CQRS
     Write Side Model
     Events
     Write to Event Log
     Commands
     

     View Slide

139. CQRS
     Write Side Model
     Events
     Events
     Write to Event Log
     Commands
     

     View Slide

140. CQRS
     Read Side Model
     Write Side Model
     Events
     Events
     Read Data Store
     Write to Event Log
     Commands
     

     View Slide

141. CQRS
     Read Side Model
     Write Side Model
     Events
     Queries
     Events
     Read Data Store
     Write to Event Log
     Commands
     

     View Slide

142. CQRS
     Read Side Model
     Write Side Model
     Events
     Queries
     Events
     Read Data Store
     Write to Event Log
     Eventual
     Consistency
     Commands
     

     View Slide

143. ✴ Temporal Decoupling
     ✴ Increased Resilience
     ✴ Increased Scalability
     ✴ Allows for Multiple Event subscribers
     ✴ Allows for Polyglot Persistence
     Advantages
     Of Using CQRS
     

     View Slide

144. ✴ More infrastructure to manage
     ✴ 2 data models (or more) to maintain
     ✴ Eventual Consistency
     Disadvantages
     Of Using CQRS
     

     View Slide

145. Events
     Allow Us To Manage
     Time
     

     View Slide

146. “Modelling events forces you to have a 

     temporal focus on what’s going on in the system.
     Time becomes a crucial factor of the system.”
     - Greg Young
     A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)
     

     View Slide

147. ✴ Event is a snapshot in time
     ✴ Event ID is an index for time
     ✴ Event Log is our full history
     The database of Our past
     The path to Our present
     Event Sourcing
     Allows Us To
     Model Time
     

     View Slide

148. Event Sourcing
     Allows For
     Time Travel
     

     View Slide

149. Event Sourcing
     Allows For
     Time Travel
     

     View Slide

150. Event Sourcing
     Allows For
     Time Travel
     ✴Replay the log for historic debugging
     ✴Replay the log for auditing & traceability
     ✴Replay the log on failure
     ✴Replay the log for replication
     

     View Slide

151. We Can Even Fork the Past
     ...Or Join Two Distinct Pasts
     

     View Slide

152. Key Takeaways
     

     View Slide

153. Key Takeaways
     Events-First design helps you to:
     ✴ reduce risk when modernizing applications
     ✴ Move Faster towards a Resilient and Scalable architecture
     ✴ Design autonomous services
     ✴ Balance Certainty and Uncertainty
     

     View Slide

154. Key Takeaways
     Events-First design helps you to:
     ✴ reduce risk when modernizing applications
     ✴ Move Faster towards a Resilient and Scalable architecture
     ✴ Design autonomous services
     ✴ Balance Certainty and Uncertainty
     Event Logging allows you to:
     ✴ AVOID CRUD and ORM
     ✴ take control of your system’s history
     ✴ time-travel
     ✴ Balance Strong and eventual consistency
     

     View Slide

155. http://akka.io
     

     View Slide

156. Learn More
     Download my latest book for free at:
     bit.ly/reactive-microsystems
     

     View Slide