Building event sourced systems with Kafka Streams

Slides from my talk "Building event sourced systems with Kafka Streams" in Codemotion Milan 2018

https://milan2018.codemotionworld.com

Transcript

1. Event Sourcing w/ Kafka Streams Amitay Horwitz | @amitayh Milan | November 29 - 30, 2018
4. AGENDA ✅
5. AGENDA ✅ • Event sourcing 101
6. AGENDA ✅ • Event sourcing 101 • Eventim
7. AGENDA ✅ • Event sourcing 101 • Eventim • Kafka & Kafka Streams
8. AGENDA ✅ • Event sourcing 101 • Eventim • Kafka & Kafka Streams • Putting it all together
9. BACKGROUND
10. A SERVICE IS BORN ✒ • Wix Invoices was incepted in mid 2015 • Rich domain model • Auditing is important for monetary products
20. invoices invoice_id customer_id issue_date due_date sent_date currency status line_items line_item_id invoice_id description quantity price customers customer_id name email address payments transaction_id invoice_id payment_type payment_amount taxes tax_id name rate NAÏVE SOLUTION tax_to_line line_item_id tax_id
22. HOW DO YOU HYDRATE YOUR DOMAIN OBJECTS? SELECT e.employee_id AS "Employee #" , e.first_name || ' ' || e.last_name AS "Name" , e.email AS "Email" , e.phone_number AS "Phone" , TO_CHAR(e.hire_date, 'MM/DD/YYYY') AS "Hire Date" , TO_CHAR(e.salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') AS "Salary" , e.commission_pct AS "Comission %" , 'works as ' || j.job_title || ' in ' || d.department_name || ' department (manager: ' || dm.first_name || ' ' || dm.last_name || ') and immediate supervisor: ' || m.first_name || ' ' || m.last_name AS "Current Job" , TO_CHAR(j.min_salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') || ' - ' || TO_CHAR(j.max_salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') AS "Current Salary" , l.street_address || ', ' || l.postal_code || ', ' || l.city || ', ' || l.state_province || ', ' || c.country_name || ' (' || r.region_name || ')' AS "Location" , jh.job_id AS "History Job ID" , 'worked from ' || TO_CHAR(jh.start_date, 'MM/DD/YYYY') || ' to ' || TO_CHAR(jh.end_date, 'MM/DD/YYYY') || ' as ' || jj.job_title || ' in ' || dd.department_name || ' department' AS "History Job Title" FROM employees e -- to get title of current job_id JOIN jobs j ON e.job_id = j.job_id -- to get name of current manager_id LEFT JOIN employees m ON e.manager_id = m.employee_id -- to get name of current department_id LEFT JOIN departments d ON d.department_id = e.department_id -- to get name of manager of current department -- (not equal to current manager and can be equal to the employee itself) LEFT JOIN employees dm ON d.manager_id = dm.employee_id -- to get name of location
23. HOW DO YOU HYDRATE YOUR DOMAIN OBJECTS? SELECT e.employee_id AS "Employee #" , e.first_name || ' ' || e.last_name AS "Name" , e.email AS "Email" , e.phone_number AS "Phone" , TO_CHAR(e.hire_date, 'MM/DD/YYYY') AS "Hire Date" , TO_CHAR(e.salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') AS "Salary" , e.commission_pct AS "Comission %" , 'works as ' || j.job_title || ' in ' || d.department_name || ' department (manager: ' || dm.first_name || ' ' || dm.last_name || ') and immediate supervisor: ' || m.first_name || ' ' || m.last_name AS "Current Job" , TO_CHAR(j.min_salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') || ' - ' || TO_CHAR(j.max_salary, 'L99G999D99', 'NLS_NUMERIC_CHARACTERS = ''.,'' NLS_CURRENCY = ''$''') AS "Current Salary" , l.street_address || ', ' || l.postal_code || ', ' || l.city || ', ' || l.state_province || ', ' || c.country_name || ' (' || r.region_name || ')' AS "Location" , jh.job_id AS "History Job ID" , 'worked from ' || TO_CHAR(jh.start_date, 'MM/DD/YYYY') || ' to ' || TO_CHAR(jh.end_date, 'MM/DD/YYYY') || ' as ' || jj.job_title || ' in ' || dd.department_name || ' department' AS "History Job Title" FROM employees e -- to get title of current job_id JOIN jobs j ON e.job_id = j.job_id -- to get name of current manager_id LEFT JOIN employees m ON e.manager_id = m.employee_id -- to get name of current department_id LEFT JOIN departments d ON d.department_id = e.department_id -- to get name of manager of current department -- (not equal to current manager and can be equal to the employee itself) LEFT JOIN employees dm ON d.manager_id = dm.employee_id -- to get name of location
24. DOMAIN MODEL ≠ DB TABLE
25. EVENT SOURCING 101
26. MUTABLE STATE
27. MUTABLE STATE • Instead of saving the current state, we save the succession of events that brought us to this state
28. MUTABLE STATE • Instead of saving the current state, we save the succession of events that brought us to this state • currentState = fold(events, emptyState)
29. INVOICE LIFECYCLE time
30. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [], "status": "DRAFT" } time
31. Invoice created { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 1.99, "qty": 1}], "status": "DRAFT" } Line item added time INVOICE LIFECYCLE
32. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 1.99, "qty": 1}, {"price": 3.50, "qty": 2}], "status": "DRAFT" } Line item added Line item added time
33. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 3.50, "qty": 2}], "status": "DRAFT" } Line item added Line item added Line item removed time
34. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 3.50, "qty": 2}], "status": "SENT" } Line item added Line item added Line item removed Invoice sent to customer time
35. Invoice created INVOICE LIFECYCLE { "customer": {...}, "issueDate": "2018-01-01", "dueDate": "2018-02-01", "lineItems: [{"price": 3.50, "qty": 2}], "status": "PAID" } Line item added Line item added Line item removed Invoice sent to customer Payment received time
36. EVENTIM
37. DESIGN GOALS • Small and simple library • Non-intrusive • Maintain data integrity • Easily add custom views
38. WRITE PATH ✏ Event Sourced
 Command Handler User Interface
39. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface
40. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface invoice ID: 12345
 expected version: 5
 amount: $12.34
 Add Payment
41. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface invoice ID: 12345
 expected version: 5
 amount: $12.34
 Add Payment Aggregare
 Repository Load
42. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface invoice ID: 12345
 expected version: 5
 amount: $12.34
 Add Payment Aggregare
 Repository Load Event
 Store Get events
43. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface invoice ID: 12345
 expected version: 5
 amount: $12.34
 Add Payment Aggregare
 Repository Load Event
 Store Get events version: 5
 customer: {...}
 line items: [...]
 balance: $12.34 Invoice #12345
44. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface Aggregare
 Repository Load Event
 Store Get events invoice ID: 12345
 expected version: 5
 amount: $12.34 Add Payment version: 5
 customer: {...}
 line items: [...]
 balance: $12.34 Invoice #12345
45. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface Aggregare
 Repository Load Event
 Store Get events invoice ID: 12345
 expected version: 5
 amount: $12.34 Add Payment version: 5
 customer: {...}
 line items: [...]
 balance: $12.34 Invoice #12345
46. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface Aggregare
 Repository Load Event
 Store Get events
47. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface Aggregare
 Repository Load Event
 Store Get events Payment Added: $12.34
48. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface Aggregare
 Repository Load Event
 Store Get events Payment Added: $12.34 Status Changed: Paid
49. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface Aggregare
 Repository Load Event
 Store Get events Payment Added: $12.34 Status Changed: Paid version: 7
 customer: {...}
 line items: [...]
 balance: $0.00 Invoice #12345
50. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface Aggregare
 Repository Load Event
 Store Get events Payment Added: $12.34 Status Changed: Paid Publish events (OCC) version: 7
 customer: {...}
 line items: [...]
 balance: $0.00 Invoice #12345
51. WRITE PATH ✏ Event Sourced
 Command Handler Command User Interface Aggregare
 Repository Load Event
 Store Get events Payment Added: $12.34 Status Changed: Paid Publish events (OCC) Event Bus
52. READ PATH Event Bus
53. READ PATH Event Bus Events
54. READ PATH View Projector Event Handler Event Bus Events
55. READ PATH View Projector Event Handler DB Event Bus Events
56. READ PATH View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $12.34 New 67890 Jane Doe $34.56 Sent
57. READ PATH View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $12.34 New 67890 Jane Doe $34.56 Sent Payment Added: $12.34
58. READ PATH View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $0.00 New 67890 Jane Doe $34.56 Sent
59. READ PATH View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $0.00 New 67890 Jane Doe $34.56 Sent Status Changed: Paid
60. READ PATH View Projector Event Handler DB Event Bus Events invoice_id customer balance status 12345 John Doe $0.00 Paid 67890 Jane Doe $34.56 Sent
61. READ PATH View Projector Event Handler DB Event Bus Events
62. READ PATH View Projector Event Handler DB Event Bus Events
63. READ PATH View Projector Event Handler DB Event Bus Events
64. READ PATH View Projector Event Handler DB Event Bus Events
65. READ PATH View Projector Event Handler DB Event Bus Events User Interface
66. READ PATH View Projector Event Handler DB Queries Event Bus Events User Interface
67. READ PATH Mailer ✉ Event Handler View Projector Event Handler DB Queries Event Bus Events User Interface
68. READ PATH Mailer ✉ Event Handler View Projector Event Handler Reporting Event Handler DB Queries Event Bus Events User Interface
69. PAIN POINTS
70. PAIN POINTS • Persisting events and publishing them is not atomic
71. PAIN POINTS • Persisting events and publishing them is not atomic • Inherent eventual consistency is not integrated in the product (read after write)
72. PAIN POINTS • Persisting events and publishing them is not atomic • Inherent eventual consistency is not integrated in the product (read after write) • Rebuilding views is a complex operation
73. REBUILDING VIEWS version event_payload
74. REBUILDING VIEWS version event_payload 1 InvoiceCreated(...)
75. REBUILDING VIEWS version event_payload 1 InvoiceCreated(...) 2 LineItemAdded(...)
76. REBUILDING VIEWS version event_payload 1 InvoiceCreated(...) 2 LineItemAdded(...) 3 LineItemAdded(...)
77. REBUILDING VIEWS version event_payload 1 InvoiceCreated(...) 2 LineItemAdded(...) 3 LineItemAdded(...) 4 InvoiceSent(...)
78. REBUILDING VIEWS SELECT *
 FROM events
 WHERE version > 0
 ORDER BY version
 LIMIT 100;
79. REBUILDING VIEWS SELECT *
 FROM events
 WHERE version > 100
 ORDER BY version
 LIMIT 100;
80. REBUILDING VIEWS invoice_id invoice_version event_payload
81. REBUILDING VIEWS invoice_id invoice_version event_payload 12345 1 InvoiceCreated
82. REBUILDING VIEWS invoice_id invoice_version event_payload 12345 1 InvoiceCreated 12345 2 LineItemAdded
83. REBUILDING VIEWS invoice_id invoice_version event_payload 12345 1 InvoiceCreated 12345 2 LineItemAdded 67890 1 InvoiceCreated
84. REBUILDING VIEWS invoice_id invoice_version event_payload 12345 1 InvoiceCreated 12345 2 LineItemAdded 67890 1 InvoiceCreated 67890 2 InvoiceDeleted
85. REBUILDING VIEWS invoice_id invoice_version event_payload 12345 1 InvoiceCreated 12345 2 LineItemAdded 12345 3 InvoiceSent 67890 1 InvoiceCreated 67890 2 InvoiceDeleted
86. REBUILDING VIEWS invoice_id version payload timestamp 12345 1 ... 14:05 12345 2 ... 14:06 12345 3 ... 15:50 67890 1 ... 15:30 67890 2 ... 15:33
87. REBUILDING VIEWS invoice_id version payload timestamp 12345 1 ... 14:05 12345 2 ... 14:06 67890 1 ... 15:30 67890 2 ... 15:33 12345 3 ... 15:50
88. REBUILDING VIEWS invoice_id version payload order 12345 1 ... 1 12345 2 ... 2 67890 1 ... 3 67890 2 ... 4 12345 3 ... 5
89. KAFKA & KAFKA STREAMS 101
90. APACHE KAFKA • Distributed append-only log • Replicated, fault-tolerant • Often used as pub-sub or queue • Used heavily at LinkedIn, Netflix, Wix and many others
91. KAFKA TOPICS 6 5 4 3 2 1 4 3 2 1 7 6 5 4 3 2 1 P0 P1 P2 Producer
92. KAFKA TOPICS Consumer Group Node #1 6 5 4 3 2 1 4 3 2 1 7 6 5 4 3 2 1 P0 P1 P2
93. KAFKA TOPICS Consumer Group Node #1 6 5 4 3 2 1 4 3 2 1 7 6 5 4 3 2 1 P0 P1 P2 Node #2
94. TOPICS RETENTION 7 6 5 4 3 2 1 Consumer #1
95. TOPICS RETENTION 7 6 5 4 3 2 1 Consumer #1 Consumer #2
96. TOPICS RETENTION 7 6 5 4 3 2 1 Consumer #1 Consumer #2
97. TOPICS RETENTION 7 6 5 4 3 2 1 Consumer #1 Consumer #2
98. KAFKA STREAMS
99. KAFKA STREAMS Streams
100. KAFKA STREAMS Streams Tables
101. STREAMS ✈ • "Data in flight" • Unbounded, continuously updating data set • Ordered, replayable, sequence of immutable data key-value pairs
102. TABLES • "Data at rest" • A collection of evolving facts • A point-in-time view of aggregated data
103. STREAM-TABLE DUALITY
104. STREAM-TABLE DUALITY
105. STREAM-TABLE DUALITY Stream of all changes to a table
106. STREAM-TABLE DUALITY Stream of all changes to a table Materialize a stream into a table
107. STREAM-TABLE DUALITY User Pageviews alice 1
108. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1
109. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1
110. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1)
111. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1)
112. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2)
113. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2) User Pageviews alice 1
114. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2) User Pageviews alice 1 User Pageviews alice 1 charlie 1
115. STREAM-TABLE DUALITY User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1 ("alice", 1) ("charlie", 1) ("alice", 2) User Pageviews alice 1 User Pageviews alice 1 charlie 1 User Pageviews alice 2 charlie 1
116. STREAM TRANSFORMATIONS
117. STREAM TRANSFORMATIONS
118. STREAM TRANSFORMATIONS
119. STREAM TRANSFORMATIONS • map
120. STREAM TRANSFORMATIONS • map • filter
121. STREAM TRANSFORMATIONS • map • filter • flatMap
122. STREAM TRANSFORMATIONS • map • filter • flatMap • …
123. STREAM TRANSFORMATIONS • map • filter • flatMap • …
124. STREAM TRANSFORMATIONS
125. STREAM TRANSFORMATIONS
126. STREAM TRANSFORMATIONS
127. STREAM TRANSFORMATIONS Stream-stream join
128. STREAM TRANSFORMATIONS
129. STREAM TRANSFORMATIONS
130. STREAM TRANSFORMATIONS
131. STREAM TRANSFORMATIONS Stream-table join
132. STREAM PROCESSING APP Streams API Your app
133. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data
134. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing
135. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing • Supports windowing operations
136. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing • Supports windowing operations • Embedded in your app
137. STREAM PROCESSING APP Streams API Your app • Transforms and enriches data • Stateless / stateful processing • Supports windowing operations • Embedded in your app • Elastic, scaleable, fault-tolerant
138. val props = new Properties props.put( StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
 "localhost:9092") props.put( StreamsConfig.APPLICATION_ID_CONFIG, "my-streams-app")
139. STREAM PROCESSING APP Streams API Your app Streams API Your app Streams API Your app
140. PROCESSOR TOPOLOGY group map join filter join Source processor Sink processor Source processor
141. PROCESSOR TOPOLOGY group map join filter join Source processor Sink processor Source processor state-store
142. PROCESSOR API • The most low-level • Interact with state-stores, schedulers, etc. • All standard operations are implemented like this (map / filter / …) • Create your own custom processing logic!
143. STREAMS DSL • Programmatically describe your topology
144. STREAMS DSL val builder = new StreamsBuilder • Programmatically describe your topology
145. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") • Programmatically describe your topology
146. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines • Programmatically describe your topology
147. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) • Programmatically describe your topology
148. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) .groupBy((_, word) => word) • Programmatically describe your topology
149. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) .groupBy((_, word) => word) .count(Materialized.as("counts-store")) • Programmatically describe your topology
150. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) .groupBy((_, word) => word) .count(Materialized.as("counts-store")) wordCounts.toStream.to("WordsWithCountsTopic") • Programmatically describe your topology
151. STREAMS DSL val builder = new StreamsBuilder val textLines: KStream[String, String] = builder.stream("TextLinesTopic") val wordCounts: KTable[String, Long] = textLines .flatMapValues(textLine => textLine.split("\\W+")) .groupBy((_, word) => word) .count(Materialized.as("counts-store")) wordCounts.toStream.to("WordsWithCountsTopic") • Programmatically describe your topology
152. KSQL • SQL dialect for streaming data CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 SECONDS) GROUP BY card_number HAVING count(*) > 3;
153. PUTTING EVERYTHING TOGETHER
154. HIGH LEVEL DESIGN User Interface
155. HIGH LEVEL DESIGN User Interface
156. HIGH LEVEL DESIGN User Interface
157. HIGH LEVEL DESIGN User Interface {"create":…}
158. HIGH LEVEL DESIGN User Interface invoice-commands
159. HIGH LEVEL DESIGN User Interface Command Handler
160. HIGH LEVEL DESIGN User Interface Command Handler Stream processing app
161. HIGH LEVEL DESIGN User Interface command-results Command Handler Web Socket
162. HIGH LEVEL DESIGN User Interface Projector Projection DB Command Handler
163. HIGH LEVEL DESIGN User Interface Projector Projection DB Command Handler
164. HIGH LEVEL DESIGN User Interface Projector Projector Command Handler Command Handler Command Handler
166. COMMAND HANDLER
167. COMMAND HANDLER commands stream invoice-commands
168. COMMAND HANDLER commands stream invoice-commands invoice ID: 12345
 command ID: 67890
 amount: $12.34
 Add Payment
169. COMMAND HANDLER commands stream invoice-commands invoice ID: 12345
 command ID: 67890
 amount: $12.34
 Add Payment
170. COMMAND HANDLER transform: results commands stream invoice-commands invoice ID: 12345
 command ID: 67890
 amount: $12.34 Add Payment
171. COMMAND HANDLER transform: results commands stream invoice-commands Snapshots state-store invoice ID: 12345
 command ID: 67890
 amount: $12.34 Add Payment
172. COMMAND HANDLER transform: results commands stream invoice-commands Snapshots state-store invoice ID: 12345
 command ID: 67890
 amount: $12.34 Add Payment customer: {...}
 line items: [...]
 balance: $12.34 Invoice #12345
173. COMMAND HANDLER transform: results commands stream invoice-commands Snapshots state-store invoice ID: 12345
 command ID: 67890
 amount: $12.34 Add Payment customer: {...}
 line items: [...]
 balance: $12.34 Invoice #12345
174. COMMAND HANDLER transform: results commands stream invoice-commands Snapshots state-store
175. COMMAND HANDLER transform: results commands stream invoice-commands Snapshots state-store Payment Added: $12.34
176. COMMAND HANDLER transform: results commands stream invoice-commands Snapshots state-store Payment Added: $12.34 Status Changed: Paid
177. COMMAND HANDLER transform: results commands stream invoice-commands Snapshots state-store Payment Added: $12.34 Status Changed: Paid customer: {...}
 line items: [...]
 balance: $0.00 Invoice #12345
178. COMMAND HANDLER command-results transform: results commands stream invoice-commands Snapshots state-store command ID: 67890
 snapshot: {...}
 events: [...] Successful
179. COMMAND HANDLER command-results transform: results commands stream invoice-commands Snapshots state-store
180. COMMAND HANDLER command-results transform: results commands stream invoice-commands Snapshots state-store command ID: 67890
 cause: invoice deleted
 
 Failed
181. COMMAND HANDLER command-results transform: results commands stream invoice-commands Snapshots state-store
182. COMMAND HANDLER command-results transform: results commands stream invoice-commands Snapshots state-store Short retention (~1 hour)
183. COMMAND HANDLER command-results filter: successful transform: results commands stream invoice-commands Snapshots state-store
184. COMMAND HANDLER command-results invoice-events filter: successful transform: results commands stream invoice-commands Snapshots state-store flatMap: events
185. COMMAND HANDLER command-results invoice-events filter: successful transform: results commands stream invoice-commands Snapshots state-store flatMap: events map: snapshots invoice-snapshots
186. COMMAND HANDLER command-results invoice-events filter: successful transform: results commands stream invoice-commands Snapshots state-store flatMap: events map: snapshots invoice-snapshots Long retention (forever) Compacted
187. PROJECTOR
188. PROJECTOR snapshots stream invoice-snapshots
189. PROJECTOR map: records snapshots stream invoice-snapshots
190. PROJECTOR foreach: persist DB map: records snapshots stream invoice-snapshots
191. PROJECTOR foreach: persist DB invoice-records map: records snapshots stream invoice-snapshots
192. API DB Business Logic Persistance Service #2 Service #3 Service #4
194. Invoices
195. Invoices Customers Orders Products
196. Invoices Inventory Customers Orders Products
197. Invoices Inventory Analytics Customers Orders Products
198. Stream processors Materialized views API Writes Reads
199. WINS
200. WINS • Simple and declarative system
201. WINS • Simple and declarative system • Atomicity - Kafka used as event-store + notification
202. WINS • Simple and declarative system • Atomicity - Kafka used as event-store + notification • Eventual consistency is handled gracefully
203. WINS • Simple and declarative system • Atomicity - Kafka used as event-store + notification • Eventual consistency is handled gracefully • Easy to add or change views
204. DEMO
205. TAKEAWAYS
206. TAKEAWAYS • Event driven systems and event sourcing can help create very flexible and scalable systems
207. TAKEAWAYS • Event driven systems and event sourcing can help create very flexible and scalable systems • Know your tradeoffs (consistency, schema evolution, debugging, error handling, …)
208. TAKEAWAYS • Event driven systems and event sourcing can help create very flexible and scalable systems • Know your tradeoffs (consistency, schema evolution, debugging, error handling, …) • Kafka & Kafka Streams are powerful tools that can be employed in many use cases
209. 4USFBN
     PO
210. RESOURCES • Demo code:
 https://github.com/amitayh/event-sourcing-kafka-streams • Event sourcing by Greg Young -
 https://youtu.be/8JKjvY4etTY • Martin Kleppmann - Is Kafka a Database?
 https://youtu.be/v2RJQELoM6Y • Kafka Streams docs - http://wix.to/00C2ADs • Blog post from Confluent - http://wix.to/Z0C2ADs
211. Q&A