Event Sourcing with Kafka Streams

Slides from my talk @ JEEConf Kiev 2018
https://jeeconf.com/program/building-event-sourced-systems-with-kafka-streams/

1. &WFOU
   4PVSDJOH XJUI
   ,BGLB
   4USFBNT Amitay Horwitz
2. ~$ whois • Software engineer @ Wix • Functional programming,
 distributed systems, TDD, … • • @amitayh on Twitter, GitHub, et al
5. • 120M < Users • 2,100 < Employees • 700 < Engineers • 600 < Micro-services in production
6. Israel !
 Tel Aviv
 Be'er Sheva Ukraine "
 Kiev
 Dnipro Lithuania #
 Vilnius
7. AGENDA ✅
8. AGENDA ✅ • Event sourcing 101
9. AGENDA ✅ • Event sourcing 101 • Eventim
10. AGENDA ✅ • Event sourcing 101 • Eventim • Kafka & Kafka Streams
11. AGENDA ✅ • Event sourcing 101 • Eventim • Kafka & Kafka Streams • Putting it all together
12. BACKGROUND
13. A SERVICE IS BORN ✒ • Wix Invoices was incepted in mid 2015 • Rich domain model • Auditing is important for monetary products
23. EVENT SOURCING 101
24. invoices invoice_id customer_id issue_date due_date sent_date currency status line_items line_item_id invoice_id description quantity price customers customer_id name email address payments transaction_id invoice_id payment_type payment_amount taxes tax_id line_item_id name rate OBJECT RELATIONAL IMPEDANCE MISMATCH
25. DOMAIN MODEL ≠ DB TABLE
26. MUTABLE STATE
27. MUTABLE STATE • Instead of saving the current state, we save the succession of events that brought us to this state
28. MUTABLE STATE • Instead of saving the current state, we save the succession of events that brought us to this state • currentState =
 events.foldLeft(empty) { (state, event) =>
 state apply event
 }
29. INVOICE LIFECYCLE time
30. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [], "status": "DRAFT" } time
31. Invoice created { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 1.99, "qty": 1}], "status": "DRAFT" } Line item added time INVOICE LIFECYCLE
32. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 1.99, "qty": 1}, {"price": 3.50, "qty": 2}], "status": "DRAFT" } Line item added Line item added time
33. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 3.50, "qty": 2}], "status": "DRAFT" } Line item added Line item added Line item removed time
34. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 3.50, "qty": 2}], "status": "SENT" } Line item added Line item added Line item removed Invoice sent to customer time
35. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 3.50, "qty": 2}], "status": "PAID" } Line item added Line item added Line item removed Invoice sent to customer Payment received time
36. EVENTIM
37. DESIGN GOALS • Small and simple library • Non-intrusive • Maintain data integrity • Easily add custom views
38. WRITE PATH ✏ User Interface
39. WRITE PATH ✏ User Interface Command JSON
40. WRITE PATH ✏ User Interface Command JSON Command DTO Decoder
41. WRITE PATH ✏ User Interface Command JSON Command DTO Decoder Validate command payload
42. WRITE PATH ✏ User Interface Command JSON Command DTO Decoder Domain Command Validate command payload
43. WRITE PATH ✏ User Interface Command JSON Command DTO Decoder Domain Command Handler #1 … Command Dispatcher Handler #2 Handler #n Validate command payload
44. WRITE PATH ✏ User Interface Command JSON Command DTO Decoder Domain Command Handler #1 … Command Dispatcher Handler #2 Handler #n Validate command payload PartialFunction[DomainCommand, Unit]
45. WRITE PATH ✏ User Interface Command JSON Command DTO Decoder Domain Command Handler #1 … Command Dispatcher Handler #2 Handler #n Validate command payload PartialFunction[DomainCommand, Unit] Function[DomainCommand, Unit]
46. WRITE PATH ✏ Event Sourced
 Command Handler
47. WRITE PATH ✏ Event Sourced
 Command Handler Command
48. WRITE PATH ✏ Event Sourced
 Command Handler Aggregare
 Repository Load Command
49. WRITE PATH ✏ Event Sourced
 Command Handler Aggregare
 Repository Load Event
 Store Get events Command
50. WRITE PATH ✏ Event Sourced
 Command Handler Aggregare
 Repository Load Event
 Store Get events Current
 aggregate Command
51. WRITE PATH ✏ Event Sourced
 Command Handler Aggregare
 Repository Load Event
 Store Get events Current
 aggregate Command
 events Command
52. WRITE PATH ✏ Event Sourced
 Command Handler Aggregare
 Repository Load Event
 Store Get events New Aggregate Current
 aggregate Command
 events Command
53. WRITE PATH ✏ Event Sourced
 Command Handler Aggregare
 Repository Load Event
 Store Get events New Aggregate Current
 aggregate Command
 events Publish events (optimistic locking) Command
54. WRITE PATH ✏ Event Sourced
 Command Handler Aggregare
 Repository Load Event
 Store Get events New Aggregate Current
 aggregate Command
 events Publish events (optimistic locking) Event Bus Command
55. READ PATH Event Bus
56. READ PATH Event Bus Events
57. READ PATH View Projector Event Handler Event Bus Events
58. READ PATH View Projector Event Handler DB Event Bus Events
59. READ PATH Mailer ✉ Event Handler View Projector Event Handler DB Event Bus Events
60. READ PATH Mailer ✉ Event Handler View Projector Event Handler Reporting Event Handler DB Event Bus Events
61. READ PATH Mailer ✉ Event Handler View Projector Event Handler Reporting Event Handler DB Event Bus Events User Interface
62. READ PATH Mailer ✉ Event Handler View Projector Event Handler Reporting Event Handler DB Queries Event Bus Events User Interface
63. PAIN POINTS • Despite of the simple design, became quite a big library • Inherent eventual consistency is not integrated in the product (read after write) • Rebuilding views is a complex operation
64. REBUILDING VIEWS Event Bus Events
65. REBUILDING VIEWS View Projector #1 Event Handler DB #1 Event Bus Events
66. REBUILDING VIEWS View Projector #1 Event Handler DB #1 Event Bus Events View Projector #2 Event Handler DB #2
67. REBUILDING VIEWS View Projector #1 Event Handler DB #1 Event Bus Events View Projector #2 Event Handler DB #2 User Interface Queries
68. KAFKA & KAFKA STREAMS 101
69. APACHE KAFKA • Distributed append-only log • Replicated, fault-tolerant • Often used as pub-sub or queue • Used heavily at LinkedIn, Netflix, Wix and many others
70. KAFKA TOPICS 6 5 4 3 2 1 4 3 2 1 7 6 5 4 3 2 1 P0 P1 P2 Producer
71. KAFKA TOPICS Consumer Group Node #1 6 5 4 3 2 1 4 3 2 1 7 6 5 4 3 2 1 P0 P1 P2
72. KAFKA TOPICS Consumer Group Node #1 6 5 4 3 2 1 4 3 2 1 7 6 5 4 3 2 1 P0 P1 P2 Node #2
73. KAFKA STREAMS
74. KAFKA STREAMS Streams
75. KAFKA STREAMS Streams Tables
76. STREAMS ✈ • "Data in flight" • Unbounded, continuously updating data set • Ordered, replayable, sequence of immutable data key-value pairs
77. TABLES • "Data at rest" • A collection of evolving facts • A point-in-time view of aggregated data
78. STREAM-TABLE DUALITY
79. STREAM-TABLE DUALITY
80. STREAM-TABLE DUALITY Stream of all changes to a table
81. STREAM-TABLE DUALITY Stream of all changes to a table Materialize a stream into a table
82. STREAM-TABLE DUALITY User Pageviews alice 1
83. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1
84. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1
85. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1)
86. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1)
87. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2)
88. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2) User Pageviews alice 1
89. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2) User Pageviews alice 1 User Pageviews alice 1 charlie 1
90. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2) User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1
91. STREAM TRANSFORMATIONS
92. STREAM TRANSFORMATIONS
93. STREAM TRANSFORMATIONS
94. STREAM TRANSFORMATIONS • map
95. STREAM TRANSFORMATIONS • map • filter
96. STREAM TRANSFORMATIONS • map • filter • flatMap
97. STREAM TRANSFORMATIONS • map • filter • flatMap • …
98. STREAM TRANSFORMATIONS • map • filter • flatMap • …
99. STREAM TRANSFORMATIONS
100. STREAM TRANSFORMATIONS
101. STREAM TRANSFORMATIONS
102. STREAM TRANSFORMATIONS Stream-stream join
103. STREAM TRANSFORMATIONS
104. STREAM TRANSFORMATIONS
105. STREAM TRANSFORMATIONS
106. STREAM TRANSFORMATIONS Stream-table join
107. STREAM PROCESSING APP Streams API Your app
108. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data
109. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing
110. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing • Supports windowing operations
111. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing • Supports windowing operations • Embedded in your app
112. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing • Supports windowing operations • Embedded in your app • Elastic, scaleable, fault-tolerant
113. val props = new Properties props.put( StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
 "localhost:9092") props.put( StreamsConfig.APPLICATION_ID_CONFIG, "my-streams-app")
114. STREAM PROCESSING APP Streams API Your app Streams API Your app Streams API Your app
115. PROCESSOR TOPOLOGY group map join filter join Source processor Sink processor Source processor
116. PROCESSOR API • The most low-level • Interact with state-stores, schedulers, etc. • All standard operations are implemented like this (map / filter / …) • Create your own custom processing logic
117. STREAMS DSL • Programmatically describe your topology
118. STREAMS DSL val builder = new StreamsBuilder • Programmatically describe your topology
119. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") • Programmatically describe your topology
120. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines • Programmatically describe your topology
121. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) • Programmatically describe your topology
122. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) .groupBy((_, word) => word) • Programmatically describe your topology
123. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) .groupBy((_, word) => word) .count(Materialized.as("counts-store")) • Programmatically describe your topology
124. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) .groupBy((_, word) => word) .count(Materialized.as("counts-store")) wordCounts.toStream.to("WordsWithCountsTopic") • Programmatically describe your topology
125. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) .groupBy((_, word) => word) .count(Materialized.as("counts-store")) wordCounts.toStream.to("WordsWithCountsTopic") • Programmatically describe your topology
126. KSQL • SQL dialect for streaming data CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 SECONDS) GROUP BY card_number HAVING count(*) > 3;
127. PUTTING EVERYTHING TOGETHER
128. HIGH LEVEL DESIGN User Interface
129. HIGH LEVEL DESIGN User Interface
130. HIGH LEVEL DESIGN User Interface
131. HIGH LEVEL DESIGN User Interface {"create":…}
132. HIGH LEVEL DESIGN User Interface invoice-commands
133. HIGH LEVEL DESIGN User Interface Command Handler
134. HIGH LEVEL DESIGN User Interface Command Handler Stream processing app
135. HIGH LEVEL DESIGN User Interface command-results Command Handler Web Socket
136. HIGH LEVEL DESIGN User Interface Projector Command Handler
137. HIGH LEVEL DESIGN User Interface Projector Projection DB Command Handler
138. HIGH LEVEL DESIGN User Interface Projector Projection DB Command Handler
139. HIGH LEVEL DESIGN User Interface Projector Projector Command Handler Command Handler Command Handler
141. COMMAND HANDLER
142. COMMAND HANDLER commands stream invoice-commands
143. COMMAND HANDLER map: results commands stream invoice-commands
144. COMMAND HANDLER map: results commands stream invoice-commands Snapshots state-store
145. COMMAND HANDLER command-results map: results commands stream invoice-commands Snapshots state-store
146. COMMAND HANDLER command-results flatMap: events map: results commands stream invoice-commands Snapshots state-store
147. COMMAND HANDLER command-results invoice-events flatMap: events map: results commands stream invoice-commands Snapshots state-store
148. COMMAND HANDLER command-results invoice-events flatMap: events map: results commands stream invoice-commands Snapshots state-store events stream
149. COMMAND HANDLER command-results invoice-events flatMap: events map: results commands stream invoice-commands aggregate: snapshots Snapshots state-store events stream
150. COMMAND HANDLER command-results invoice-events flatMap: events map: results commands stream invoice-commands invoice-snapshots aggregate: snapshots Snapshots state-store events stream
151. COMMAND HANDLER command-results invoice-events flatMap: events map: results commands stream invoice-commands invoice-snapshots aggregate: snapshots Snapshots state-store events stream
152. COMMAND HANDLER val builder = new StreamsBuilder val snapshots: KTable[UUID, Snapshot[Invoice]] = builder .stream("invoice-events") .groupByKey() .aggregate( reducer.initializer, reducer.aggregator, Materialized.as("snapshots-store"))
153. PROJECTOR
154. PROJECTOR snapshots stream invoice-snapshots
155. PROJECTOR map: records snapshots stream invoice-snapshots
156. PROJECTOR foreach: persist DB map: records snapshots stream invoice-snapshots
157. PROJECTOR foreach: persist DB invoice-records map: records snapshots stream invoice-snapshots
159. Invoices
160. Invoices Customers Orders Products
161. Invoices Inventory Customers Orders Products
162. Invoices Inventory Analytics Customers Orders Products
163. WINS
164. WINS • Simple and declarative system
165. WINS • Simple and declarative system • Eventual consistency is handled gracefully
166. WINS • Simple and declarative system • Eventual consistency is handled gracefully • Easy to add or change views
167. WINS • Simple and declarative system • Eventual consistency is handled gracefully • Easy to add or change views • Benefits of event sourcing + scalability and fault- tolerance properties of Kafka
168. DEMO
169. TAKEAWAYS
170. TAKEAWAYS • Event driven systems and event sourcing can help create very flexible and scaleable systems
171. TAKEAWAYS • Event driven systems and event sourcing can help create very flexible and scaleable systems • Know your tradeoffs (consistency guarantees,
 schema evolution, data integrity, …)
172. TAKEAWAYS • Event driven systems and event sourcing can help create very flexible and scaleable systems • Know your tradeoffs (consistency guarantees,
 schema evolution, data integrity, …) • Kafka & Kafka Streams are powerful tools that can be employed in many use cases
173. 4USFBN
     PO
174. RESOURCES • Demo code:
 https://github.com/amitayh/event-sourcing-kafka-streams • Event sourcing by Greg Young -
 https://youtu.be/8JKjvY4etTY • Kafka Streams - http://wix.to/00C2ADs • Blog post from Confluent - http://wix.to/Z0C2ADs
175. Q&A