"I need my data and a little bit of your data." - Integrating services with Apache Kafka (Confluent Streaming Event Munich)

We split our monoliths into services. Every service has it's own bounded context with it's own data in its own datastore. But at some point, to do its job, the service needs additional data from other services or datastores. I show you, how to integrate them with Apache Kafka and make parts reactive.
All integrated and reactive everywhere!

1. Up Next: “I need my data and a little bit of your data.”: Integrating services with Apache Kafka (CSE Edition) René Kerner, Senior Consultant at codecentric since 06/2018, former Software-Engineer/Architect at trivago 11/2011 - 06/2018 Mail: rene.kerner@codecentric.de || Twitter: @rk3rn3r https://lparchive.org/The-Secret-of-Monkey-Island/Update%201/1-somi_001.gif
2. René Kerner 2 - >9 years Software-Engineer and Software-Architect - ~2 years of experience with Apache Kafka, CDC, Protobuf, Data Streaming - 6,5 yrs Software-Engineer/-Architect at trivago - since June 2018 Senior Consultant at codecentric - Software-Architecture, Distributed Systems - Streaming, Reactive Systems - Cloud, DevOps, Ops, Virtualisation - Development: Java, Kotlin, Spring Frame- work, PHP, Bash @rk3rn3r
3. 3 3 "I need my data and a little bit of your data." Integrating Services with Apache Kafka 3
4. Integrating Services with Apache Kafka 4 “I need my data and a little bit of your data.” - Tim Berglund, confluent
5. My Data - Your Data… huh?! 5 Imagine a web e-commerce shop architecture - Web UI / GUI - Web backend - Services for - Products - Orders - Inventory/stock - Payment - Shipping
6. Example: Dependencies of a web shop in a service world 6 - Web UI / GUI talks to its backend service
7. Web Shop Backend Dependencies 7 - Web UI / GUI talks to its backend service - The backend needs to talk to a lot of other services
8. Orders Service Dependencies 8 - Web UI / GUI talks to its backend service - The backend needs to talk to a lot of other services - The order service needs some more data too...
9. Products Service Dependencies 9 - Web UI / GUI talks to its backend service - The backend needs to talk to a lot of other services - The order service needs some more data too… - Product service might fetch stocks from stock service
10. Payment checks stocks and shipping 10 - Web UI / GUI talks to its backend service - The backend needs to talk to a lot of other services - The order service needs some more data too… - Product service might fetch stocks from stock service - Payment checks stock and triggers shipping
11. Shipping decreases stock 11 - Web UI / GUI talks to its backend service - The backend needs to talk to a lot of other services - The order service needs some more data too… - Product service might fetch stocks from stock service - Payment checks stock and triggers shipping - Shipping decreases stock
12. Example: Dependencies of a Web Shop 12 - DONE! - But we left out - Accounting - SEO/SEM - Email/Connectivity - Business Intelligence - and, and, and, … → Loooots of dependencies! - Services need data from other services - Dependencies on data and schema (and maybe behavior)
13. https://i.imgflip.com/1d7btz.jpg
14. First try... 14 Let’s build one or two services...
15. First try... 15 Let’s build one or two services... ...that use a central database!
16. Database Integration 16
17. Database Integration 17 Doesn’t look that bad!
18. Database Integration and it’s Evolution 18 growth & time natural growth
19. 19 https://i.kinja-img.com/gawker-media/image/upload/s--wl85UrLl--/c_fill,f_auto ,fl_progressive,g_center,h_675,q_80,w_1200/i4fvyeoxdyxc4pdwxaob.jpg
20. Second try... 20 Let’s do it like NETFLIX...
21. Second try... 21 Let’s do it like NETFLIX... ...HTTP APIs for the win!
22. The Wannabe-Netflix Way / HTTP APIs 22 - Very flexible, but with a lot of technical dependencies - Cascading requests, traffic and failures - Availability for 500 services: → 99,9% ^ 500 = 0,0657% - Upstream performance needs (availability, speed) are put on downstream services - sometimes multiplied
23. HTTP APIs in distributed systems are HARD! 23
24. HTTP APIs in distributed systems are HARD! 24 Optimistic Lock, Pessimistic Lock, MVCC, Session Pinning, Server Stickyness
25. 25 http://screencrush.com/files/2015/08/Patrick-Stewart-Star-Trek.jpg
26. https://i.imgflip.com/1d7btz.jpg
27. 27 Share your data. https://i.ytimg.com/vi/8eDYVtPwWiM/maxresdefault.jpg
28. 28 Share your data. But without a central database or centrally managed integration pattern! https://i.ytimg.com/vi/8eDYVtPwWiM/maxresdefault.jpg
29. Kafka Integration Scenarios 29 - Direct Integration - Producer → Producer API - Consumer → Consumer API - Data Replication - Connector → Connect API - Sink Connector - Source Connector - Reactive Data Transformation - Stream Processor (SP) → Processor API → Streams API - KSQL on top of Streams API
30. Kafka Integration Scenarios 30 - Direct Integration - Producer → Producer API - Consumer → Consumer API - Data Replication - Connector → Connect API - Sink Connector - Source Connector - Reactive Data Transformation - Stream Processor (SP) → Processor API → Streams API - KSQL on top of Streams API
31. Kafka Integration Scenarios 31 - Direct Integration - Producer → Producer API - Consumer → Consumer API - Data Replication - Connector → Connect API - Sink Connector - Source Connector - Reactive Data Transformation - Stream Processor (SP) → Processor API → Streams API - KSQL on top of Streams API
32. Step 1: Split up old DB or HTTP Integration 32 What can we do here, instead of HTTP? http://cdn.onlinewebfonts.co m/svg/img_494692.png
33. Step 1: Replicate! 33 Instead of HTTP request to fetch data, capture all changes of the “New Orders” DB table and replicate it into the datastore of the Stock service.
34. Step 1: Replicate data using Kafka Connect API 34 Instead of HTTP request to fetch data, capture all changes of the “New Orders” DB table and replicate it into the datastore of the Stock service. Kafka Topic
35. Step 1: Replicate data using Kafka Connect API 35 You could, maybe, use Change Data Capturing (CDC) with Debezium. Kafka Topic
36. Change Data Capturing (CDC) and Debezium (DBZ) 36 - Classical: Mark changed rows - hard to handle Primary Key changes and Deletes - Modern: Capture datastore’s Changelog / Commitlog / Replicationlog - Debezium - supports MySQL - supports PostgreSQL - supports MongoDB - supports Oracle DB - alpha: SQL Server 2016 SP1+ - supports Deletes and PK-changes - can handle DDL changes
37. Make use of Apache Kafka 37 Kafka Connect Kafka Connect
38. That’s it! 38 Services Integrated! We’re done. Thank you!
39. That’s it! 39 But let’s see, if we can improve!
40. DB lookups from Stock service 40 We still kept the database lookup scenarios. Maybe our Stock service recalculates stocks every few hours/minutes. A common scenario (batch, …).
41. Is there more? 41 But do we need that?
42. Can we improve? 42 When our stock goes too low, wouldn’t it be cool to directly place a new order?
43. Can we directly process our new state? 43 When our stock goes too low, wouldn’t it be cool to directly place a new order? → Directly calculate our new state
44. Step 1: Reactive! 44 Yes! Let’s make it reactive!
45. How to make Stock service reactive? 45 But how?
46. Step 1: Reactive! 46 Make your service a Kafka consumer using the Kafka Consumer API
47. Step 2: Make the service reactive using Kafka Consumer API 47 On every New Order event/dataset our business logic will be triggered to lookup and update stock. When stock is too low, we can directly place the order.
48. And another batch job... 48 Hmmm… This Recommendation service is just a cronjob that updates recommendations for the different categories once a day. At night… Can’t we?
49. Step 3: Update recommendations in a reactive way 49 Use Streams API to recalculate recommendations weights after each order. Sink them into the recommendations database table with a Sink Connector.
50. 50 Engage! https://cdn1.tvnz.co.nz/content/dam/images/news/2018/08 /05/picard.jpg.hashed.7bdfa2df.desktop.story.inline.jpg
51. Step 4: Send orders to Orders service 51
52. Step 4: Send orders to Orders service 52 - Orders service doesn’t need some of the fields of the “New Orders” messages - Some data need to be processed before they are stored
53. Step 4: Transform, Filter and Pre-Process Messages 53 - Orders service doesn’t need some of the fields of the “New Orders” messages - Some data need to be processed before they are stored → use Kafka Streams API → to transform messages and remove unneccessary fields → to make precalculations or filter → store result in a new Kafka topic and sink it into the datastore
54. Step 5: Move forward! 54
55. Architectural Pattern: Kafka Integration 55 - Simple and easy, real decoupled architecture - Many problems of a distributed system are handled - Real decoupling of producers and consumers - Producer’s work is done after ACK - Consumers are free to do whatever they want - Supports independent teams by decoupling readers and writers → CQRS, democratizing data
56. When to use Connect API? 56 If you connect to an external system that’s not directly able to connect natively to Kafka. Or, if you want to keep your well-known datastore access/lookup behavior. (e.g. legacy applications)
57. When to use Connect API? 57 If your application data access scenario is natively a lookup scenario. Incoming messages doesn’t necessarily change the application state.
58. When to use Consumer API? 58 When every message that comes in must trigger your business logic or is supposed to update your application state. e.g. Stream Processors, Reactive Dashboards, “realtime stuff”
59. When to use Streams API? 59 When you are going to write an application that consumes, processes and produces.
60. When to use a database? 60 When your application or service natively needs a lookup scenario. For example: a user wants to see all products of a specific type.
61. Kafka Usage And Benefits 61
62. 62 How to actually do this?
63. 63 63 codecentric helps you! We build, we consult, we enable: https://www.codecentric.de 63
64. 64 The future is coming.... https://vignette.wikia.nocookie.net/memoryalpha/images/9/9d/USS_Enterprise-D.jpeg/ revision/latest?cb=20131125162836&path-prefix=de
65. 65 65 Thank you! 65
66. Q & A 66 https://muppetmindset.files.wordpress.com/2016/05/question-mark.jpg?w=70 0 http://indigo.ie/~rdshiels/monkey/guyswing.gif