Using Kafka to Discover Events Hidden in your Database

Slide 1

Slide 1 text

Anna McDonald Twitter: @jbfletch_ <- fully committed to the underscore GitHub: jbfletch 1

Slide 2

Slide 2 text

ARE YOU IN THE RIGHT TALK? Does your environment look like... Super Fancy Half Fancy help. 2 @jbfletch_

Slide 3

Slide 3 text

Event Driven Overview 3 @jbfletch_

Slide 4

Slide 4 text

Event Notification 4 @jbfletch_

Slide 5

Slide 5 text

Event Carried State Transfer 5 @jbfletch_

Slide 6

Slide 6 text

Event Sourcing Fundamental Test: Do you have the ability to blow away the current application state and rebuild from an event store? 6 @jbfletch_

Slide 7

Slide 7 text

A Tale of Two Types of Events Primary Derivative CC Image courtesy of Francoise on Flickr CC Image courtesy of OtterBox on Flickr 7 @jbfletch_

Slide 8

Slide 8 text

Primary Events Derivative Events Services are Easy to Deploy Requires 4 live chickens and a complicated interpretive dance to be granted safe passage to deploy One Service per Function, No duplication Shadow Clone Jutsu! Code changes can be modeled as: {Easy to Make, Easy to Test, Easy to push} Code changes can be modeled as: {Complicated to Make, Brutal to Test, Bureaucratic to push} 8 @jbfletch_

Slide 9

Slide 9 text

I promised you mold. 9 @jbfletch_

Slide 10

Slide 10 text

Prepare to go eventing 10 @jbfletch_

Slide 11

Slide 11 text

“To find events one must know what to look for” me talking to myself, on August 27th, 2019 Know the Events that matter 11 @jbfletch_

Slide 12

Slide 12 text

“To find events one must know what to look for” me talking to myself, on August 27th, 2019 Know your Systems Know the Events that matter 12 @jbfletch_

Slide 13

Slide 13 text

“To find events one must know what to look for” me talking to myself, on August 27th, 2019 Know your System Know the Events that matter Capture Broad Categories of Events 13 @jbfletch_

Slide 14

Slide 14 text

Derivative Events Derivative eventing is quite literally deriving events from something that has been observed. 14 @jbfletch_

Slide 15

Slide 15 text

Observe Find a Durable Event Source Define an Event Profile System Logs DB User Logs 1 2 WANTED 15 @jbfletch_

Slide 16

Slide 16 text

Flashback! mySQL Debezium FREEDOM 16 The truth is in the log based CDC @jbfletch_

Slide 17

Slide 17 text

CDC Overview MySQL inserts: op_type: I, after struct Debezium 17 @jbfletch_

Slide 18

Slide 18 text

CDC Overview MySQL inserts: op_type: I, after struct updates: op_type: U, before,after struct Debezium 18 @jbfletch_

Slide 19

Slide 19 text

CDC Overview MySQL inserts: op_type: I, after struct updates: op_type: U, before,after struct deletes: op_type: D, before struct Plus optional tombstone message Debezium 19 @jbfletch_

Slide 20

Slide 20 text

Observe Find a Durable Event Source Define an Event Profile System Logs DB User Logs 1 2 WANTED 20 @jbfletch_

Slide 21

Slide 21 text

Becoming an event profiler Trigger the event in the source system Review the CDC Messages generated during the event Find the event fingerprint that signifies completeness WANTED 21 @jbfletch_

Slide 22

Slide 22 text

Becoming an event profiler Trigger the event in the source system Review the CDC Messages generated during the event Find the event fingerprint that signifies completeness WANTED 22 @jbfletch_

Slide 23

Slide 23 text

Becoming an event profiler Trigger the event in the source system Review the CDC Messages generated during the event Find the event fingerprint that signifies completeness WANTED 23 @jbfletch_

Slide 24

Slide 24 text

Example Profiles signifying completeness Simple order placed event: ● Op_type of insert, after has non null order_number 24 @jbfletch_

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Example Profiles signifying completeness Simple order placed event: ● Op_type of insert, after has non null order_number ● Op_type of update, before has a null order_number, after has a non null order number ● Op_type of update, before has a null crazy_random_field, after has a non null crazy_random_field and a non null order number 26 @jbfletch_

Slide 27

Slide 27 text

Complex Events Multiple Tables Multiple Sources MySQL Oracle What constitutes event completeness? 27 @jbfletch_

Slide 28

Slide 28 text

Order and Items example Order # 42 Total # Items 3 28 @jbfletch_

Slide 29

Slide 29 text

29 Flow Diagram @jbfletch_

Slide 30

Slide 30 text

Plan of attack 30 We need both of these things to be true before we fire the event: Order # 42 Total # Items 3 Order Event Aggregate @jbfletch_

Slide 31

Slide 31 text

How do we wait for all items to arrive? 31 @jbfletch_

Slide 32

Slide 32 text

What is the minimum information we need to know to be able to determine event completeness for items? 32 @jbfletch_ Total Number of items per order

Slide 33

Slide 33 text

33 Flow Diagram @jbfletch_

Slide 34

Slide 34 text

Basic Central Event Service Setup Filter using the event profile: op_type = “I” to create: KTable orderTableKeyOrderId <- Orders that match our order created event profile KStream itemsKeyedByItemIdStream <- Items that match our order created event profile KTable totalNumberofItemsTable <- Number of items in each order 34 1 2 3

Slide 35

Slide 35 text

groupBy + aggregate + join + filter KTable> preItemsTable = itemsKeyedByItemIdStream(ItemId,Items) .groupBy(ORDER_ID) ← OrderId: 42 .aggregate(ArrayList::new, add(Items), return null for TS) .join(totalNumberofItemsTable) <- itemCount: 3 Optional join allows for the propagation of total order items to each item 35 1 2 3 @jbfletch_

Slide 36

Slide 36 text

groupBy + aggregate + join + filter KTable> preItemsTable = itemsKeyedByItemIdStream .groupBy(ORDER_ID) ← OrderId: 42 .aggregate(ArrayList::new, add(Items), return null for TS) .join(totalNumberofItemsTable) <- itemCount: 3 Optional join allows for the propagation of total order items to each item 36 1 2 3 @jbfletch_

Slide 37

Slide 37 text

Slide 38

Slide 38 text

groupBy + aggregate + join + filter KTable> fullItemsTable = preItemsTable .filter((k,v)-> v.size() == v.get(0).get(“itemCount”).asInt()) <- This filter will block until all 3 item lines are in the array 38 4 @jbfletch_

Slide 39

Slide 39 text

Part Deux 39 How can we be sure the order message has arrived? Order # 42 Total # Items 3 Order Event Aggregate @jbfletch_

Slide 40

Slide 40 text

Thou shall not pass!!! Using inner joins 40 KTable fullOrderTable = orderTableKeyOrderId .join(fullItemsTable, <- only contains orders with all items arrived (orderNode, itemNodes) -> { construct and return order placed event aggregate }) @jbfletch_

Slide 41

Slide 41 text

Step 2 41 @jbfletch_

Slide 42

Slide 42 text

Event Schemas What do I want to tell others and what information do I need to do it? 42 @jbfletch_

Slide 43

Slide 43 text

Example Event Schema Properties Event Name - “order placed” 43 @jbfletch_

Slide 44

Slide 44 text

Example Event Schema Properties Event Name - “order placed” Event Production Time - 2019-10-16 17:37:20.000534 44 @jbfletch_

Slide 45

Slide 45 text

Example Event Schema Properties Event Name - “order placed” Event Production Time - 2019-10-16 17:37:20.000534 Database Operation Time - 2019-10-16 17:34:20.000534 45 @jbfletch_

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Example Event Schema Properties Event Name - “order placed” Event Production Time - 2019-10-16 17:37:20.000534 Database Operation Time - 2019-10-16 17:34:20.000534 Source System - “old evil application” Event Creation System - “central order event service” 47 @jbfletch_

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Step 3 49 @jbfletch_

Slide 50

Slide 50 text

Emit Go big you can always filter or branch later 50 @jbfletch_

Slide 51

Slide 51 text

Demo Code https://github.com/jbfletch/kstreams -des-demo 51 @jbfletch_

Slide 52

Slide 52 text

Presenter Info Anna McDonald @jbfletch_ Principal Software Developer, SAS Straight up Math, Event Sourcing, Apache Kafka, Integration Architecture 52