Upgrade to Pro — share decks privately, control downloads, hide ads and more …

"I need my data and a little bit of your data." - Integrating services with Apache Kafka (Confluent Streaming Event Munich)

"I need my data and a little bit of your data." - Integrating services with Apache Kafka (Confluent Streaming Event Munich)

We split our monoliths into services. Every service has it's own bounded context with it's own data in its own datastore. But at some point, to do its job, the service needs additional data from other services or datastores. I show you, how to integrate them with Apache Kafka and make parts reactive.
All integrated and reactive everywhere!

447fa14ac6e64700db87aec2bc4ddcdd?s=128

René Kerner

October 09, 2018
Tweet

Transcript

  1. Up Next: “I need my data and a little bit

    of your data.”: Integrating services with Apache Kafka (CSE Edition) René Kerner, Senior Consultant at codecentric since 06/2018, former Software-Engineer/Architect at trivago 11/2011 - 06/2018 Mail: rene.kerner@codecentric.de || Twitter: @rk3rn3r https://lparchive.org/The-Secret-of-Monkey-Island/Update%201/1-somi_001.gif
  2. René Kerner 2 - >9 years Software-Engineer and Software-Architect -

    ~2 years of experience with Apache Kafka, CDC, Protobuf, Data Streaming - 6,5 yrs Software-Engineer/-Architect at trivago - since June 2018 Senior Consultant at codecentric - Software-Architecture, Distributed Systems - Streaming, Reactive Systems - Cloud, DevOps, Ops, Virtualisation - Development: Java, Kotlin, Spring Frame- work, PHP, Bash @rk3rn3r
  3. 3 3 "I need my data and a little bit

    of your data." Integrating Services with Apache Kafka 3
  4. Integrating Services with Apache Kafka 4 “I need my data

    and a little bit of your data.” - Tim Berglund, confluent
  5. My Data - Your Data… huh?! 5 Imagine a web

    e-commerce shop architecture - Web UI / GUI - Web backend - Services for - Products - Orders - Inventory/stock - Payment - Shipping
  6. Example: Dependencies of a web shop in a service world

    6 - Web UI / GUI talks to its backend service
  7. Web Shop Backend Dependencies 7 - Web UI / GUI

    talks to its backend service - The backend needs to talk to a lot of other services
  8. Orders Service Dependencies 8 - Web UI / GUI talks

    to its backend service - The backend needs to talk to a lot of other services - The order service needs some more data too...
  9. Products Service Dependencies 9 - Web UI / GUI talks

    to its backend service - The backend needs to talk to a lot of other services - The order service needs some more data too… - Product service might fetch stocks from stock service
  10. Payment checks stocks and shipping 10 - Web UI /

    GUI talks to its backend service - The backend needs to talk to a lot of other services - The order service needs some more data too… - Product service might fetch stocks from stock service - Payment checks stock and triggers shipping
  11. Shipping decreases stock 11 - Web UI / GUI talks

    to its backend service - The backend needs to talk to a lot of other services - The order service needs some more data too… - Product service might fetch stocks from stock service - Payment checks stock and triggers shipping - Shipping decreases stock
  12. Example: Dependencies of a Web Shop 12 - DONE! -

    But we left out - Accounting - SEO/SEM - Email/Connectivity - Business Intelligence - and, and, and, … → Loooots of dependencies! - Services need data from other services - Dependencies on data and schema (and maybe behavior)
  13. https://i.imgflip.com/1d7btz.jpg

  14. First try... 14 Let’s build one or two services...

  15. First try... 15 Let’s build one or two services... ...that

    use a central database!
  16. Database Integration 16

  17. Database Integration 17 Doesn’t look that bad!

  18. Database Integration and it’s Evolution 18 growth & time natural

    growth
  19. 19 https://i.kinja-img.com/gawker-media/image/upload/s--wl85UrLl--/c_fill,f_auto ,fl_progressive,g_center,h_675,q_80,w_1200/i4fvyeoxdyxc4pdwxaob.jpg

  20. Second try... 20 Let’s do it like NETFLIX...

  21. Second try... 21 Let’s do it like NETFLIX... ...HTTP APIs

    for the win!
  22. The Wannabe-Netflix Way / HTTP APIs 22 - Very flexible,

    but with a lot of technical dependencies - Cascading requests, traffic and failures - Availability for 500 services: → 99,9% ^ 500 = 0,0657% - Upstream performance needs (availability, speed) are put on downstream services - sometimes multiplied
  23. HTTP APIs in distributed systems are HARD! 23

  24. HTTP APIs in distributed systems are HARD! 24 Optimistic Lock,

    Pessimistic Lock, MVCC, Session Pinning, Server Stickyness
  25. 25 http://screencrush.com/files/2015/08/Patrick-Stewart-Star-Trek.jpg

  26. https://i.imgflip.com/1d7btz.jpg

  27. 27 Share your data. https://i.ytimg.com/vi/8eDYVtPwWiM/maxresdefault.jpg

  28. 28 Share your data. But without a central database or

    centrally managed integration pattern! https://i.ytimg.com/vi/8eDYVtPwWiM/maxresdefault.jpg
  29. Kafka Integration Scenarios 29 - Direct Integration - Producer →

    Producer API - Consumer → Consumer API - Data Replication - Connector → Connect API - Sink Connector - Source Connector - Reactive Data Transformation - Stream Processor (SP) → Processor API → Streams API - KSQL on top of Streams API
  30. Kafka Integration Scenarios 30 - Direct Integration - Producer →

    Producer API - Consumer → Consumer API - Data Replication - Connector → Connect API - Sink Connector - Source Connector - Reactive Data Transformation - Stream Processor (SP) → Processor API → Streams API - KSQL on top of Streams API
  31. Kafka Integration Scenarios 31 - Direct Integration - Producer →

    Producer API - Consumer → Consumer API - Data Replication - Connector → Connect API - Sink Connector - Source Connector - Reactive Data Transformation - Stream Processor (SP) → Processor API → Streams API - KSQL on top of Streams API
  32. Step 1: Split up old DB or HTTP Integration 32

    What can we do here, instead of HTTP? http://cdn.onlinewebfonts.co m/svg/img_494692.png
  33. Step 1: Replicate! 33 Instead of HTTP request to fetch

    data, capture all changes of the “New Orders” DB table and replicate it into the datastore of the Stock service.
  34. Step 1: Replicate data using Kafka Connect API 34 Instead

    of HTTP request to fetch data, capture all changes of the “New Orders” DB table and replicate it into the datastore of the Stock service. Kafka Topic
  35. Step 1: Replicate data using Kafka Connect API 35 You

    could, maybe, use Change Data Capturing (CDC) with Debezium. Kafka Topic
  36. Change Data Capturing (CDC) and Debezium (DBZ) 36 - Classical:

    Mark changed rows - hard to handle Primary Key changes and Deletes - Modern: Capture datastore’s Changelog / Commitlog / Replicationlog - Debezium - supports MySQL - supports PostgreSQL - supports MongoDB - supports Oracle DB - alpha: SQL Server 2016 SP1+ - supports Deletes and PK-changes - can handle DDL changes
  37. Make use of Apache Kafka 37 Kafka Connect Kafka Connect

  38. That’s it! 38 Services Integrated! We’re done. Thank you!

  39. That’s it! 39 But let’s see, if we can improve!

  40. DB lookups from Stock service 40 We still kept the

    database lookup scenarios. Maybe our Stock service recalculates stocks every few hours/minutes. A common scenario (batch, …).
  41. Is there more? 41 But do we need that?

  42. Can we improve? 42 When our stock goes too low,

    wouldn’t it be cool to directly place a new order?
  43. Can we directly process our new state? 43 When our

    stock goes too low, wouldn’t it be cool to directly place a new order? → Directly calculate our new state
  44. Step 1: Reactive! 44 Yes! Let’s make it reactive!

  45. How to make Stock service reactive? 45 But how?

  46. Step 1: Reactive! 46 Make your service a Kafka consumer

    using the Kafka Consumer API
  47. Step 2: Make the service reactive using Kafka Consumer API

    47 On every New Order event/dataset our business logic will be triggered to lookup and update stock. When stock is too low, we can directly place the order.
  48. And another batch job... 48 Hmmm… This Recommendation service is

    just a cronjob that updates recommendations for the different categories once a day. At night… Can’t we?
  49. Step 3: Update recommendations in a reactive way 49 Use

    Streams API to recalculate recommendations weights after each order. Sink them into the recommendations database table with a Sink Connector.
  50. 50 Engage! https://cdn1.tvnz.co.nz/content/dam/images/news/2018/08 /05/picard.jpg.hashed.7bdfa2df.desktop.story.inline.jpg

  51. Step 4: Send orders to Orders service 51

  52. Step 4: Send orders to Orders service 52 - Orders

    service doesn’t need some of the fields of the “New Orders” messages - Some data need to be processed before they are stored
  53. Step 4: Transform, Filter and Pre-Process Messages 53 - Orders

    service doesn’t need some of the fields of the “New Orders” messages - Some data need to be processed before they are stored → use Kafka Streams API → to transform messages and remove unneccessary fields → to make precalculations or filter → store result in a new Kafka topic and sink it into the datastore
  54. Step 5: Move forward! 54

  55. Architectural Pattern: Kafka Integration 55 - Simple and easy, real

    decoupled architecture - Many problems of a distributed system are handled - Real decoupling of producers and consumers - Producer’s work is done after ACK - Consumers are free to do whatever they want - Supports independent teams by decoupling readers and writers → CQRS, democratizing data
  56. When to use Connect API? 56 If you connect to

    an external system that’s not directly able to connect natively to Kafka. Or, if you want to keep your well-known datastore access/lookup behavior. (e.g. legacy applications)
  57. When to use Connect API? 57 If your application data

    access scenario is natively a lookup scenario. Incoming messages doesn’t necessarily change the application state.
  58. When to use Consumer API? 58 When every message that

    comes in must trigger your business logic or is supposed to update your application state. e.g. Stream Processors, Reactive Dashboards, “realtime stuff”
  59. When to use Streams API? 59 When you are going

    to write an application that consumes, processes and produces.
  60. When to use a database? 60 When your application or

    service natively needs a lookup scenario. For example: a user wants to see all products of a specific type.
  61. Kafka Usage And Benefits 61

  62. 62 How to actually do this?

  63. 63 63 codecentric helps you! We build, we consult, we

    enable: https://www.codecentric.de 63
  64. 64 The future is coming.... https://vignette.wikia.nocookie.net/memoryalpha/images/9/9d/USS_Enterprise-D.jpeg/ revision/latest?cb=20131125162836&path-prefix=de

  65. 65 65 Thank you! 65

  66. Q & A 66 https://muppetmindset.files.wordpress.com/2016/05/question-mark.jpg?w=70 0 http://indigo.ie/~rdshiels/monkey/guyswing.gif