Save 37% off PRO during our Black Friday Sale! »

Don’t Fear Outdated Caches - Change Data Capture to the Rescue

Don’t Fear Outdated Caches - Change Data Capture to the Rescue

Joint presentation with Katia Aresti at Bordeaux JUG

The saying goes that there are only two hard things in Computer Science: cache invalidation and naming things.

Well, turns out the first is solved actually ;)

Come and join us for a session to hear not about one, but two popular open-source projects.

First, Infinispan, an in-memory distributed data store and cache, which is elastically scalable, highly availability and fault-tolerant.

Then, Debezium, a change data capture (CDC) platform for a variety of databases, such as MySQL, PostgreSQL and MongoDB.

After an overview of the key use cases and capabilities of the two projects, we'll explore in a demo how you can combine Debezium and Infinispan for even greater awesomeness:

* How to keep your cache automatically up-to-date after data changes via Debezium and CDC
* How to read denormalized views of your data with plain key look-ups from an Infinispan cluster close by
* How to propagate cache updates between clusters using cross-site replication

All this driven by Quarkus and connected via Apache Kafka. Bring your questions and ideas so we can discuss and exchange about the potential of these open-source projects in the data space!

8e25c0ca4bf25113bd9c0ccc5d118164?s=128

Gunnar Morling

January 21, 2021
Tweet

Transcript

  1. Don’t Fear Outdated Caches - Change Data Capture to the

    Rescue Discovering Infinispan and Debezium Gunnar Morling Debezium Katia Aresti Infinispan January 2021 - Home Edition Image © Nathalie https://flic.kr/p/21Ghf2g (CC BY 2.0)
  2. @karesti | @gunnarmorling #Infinispan #Debezium ...Multi-site application with shared system-of-record

    database ...With local, denormalized read views (CQRS) ... Automatically kept in sync after writes Today’s Mission Explore how to build a... 🤔
  3. @karesti | @gunnarmorling #Infinispan #Debezium Who are we? Gunnar Morling

    Katia Aresti
  4. @karesti | @gunnarmorling #Infinispan #Debezium Agenda Demo! Learn Learn

  5. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

  6. @karesti | @gunnarmorling #Infinispan #Debezium App Data Local In memory

    Cache Embedding a library
  7. @karesti | @gunnarmorling #Infinispan #Debezium App Data Local In memory

    Infinispan Local cache
  8. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 2, Maria 1,

    Juan App-1 Data App-2 Data App-3 Data Local Local Local PUT 2,Maria GET 2 REMOVE 1 Null
  9. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 2, Maria 1,

    Juan 2, Maria 1, Juan 2, Maria App-1 Data App-2 Data App-3 Data Replicated
  10. @karesti | @gunnarmorling #Infinispan #Debezium Potter-3 Spells PUT 3,Mark 3,

    Mark 3, Mark Replicated 1, Juan 2, Maria 1, Juan 2, Maria 1, Juan 2, Maria App-1 Data App-2 Data App-3 Data
  11. @karesti | @gunnarmorling #Infinispan #Debezium Replicated 1, Juan 2, Maria

    3, Mark 1, Juan 2, Maria 3, Mark 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data App-3 Data
  12. @karesti | @gunnarmorling #Infinispan #Debezium Distributed 1, Juan 2, Maria

    3, Mark App-1 Data App-2 Data App-3 Data
  13. @karesti | @gunnarmorling #Infinispan #Debezium GET 3 Mark GET 3

    Mark Distributed 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data App-3 Data
  14. @karesti | @gunnarmorling #Infinispan #Debezium 3, Reducto App-3 Data Distributed

    Null GET 3 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data
  15. @karesti | @gunnarmorling #Infinispan #Debezium PUT 4, Bella 4, Bella

    Distributed 1, Juan 3, Mark 2, Maria 1, Juan 3, Mark 2, Maria App-1 Data App-2 Data App-3 Data
  16. @karesti | @gunnarmorling #Infinispan #Debezium Distributed 1, Juan 3, Mark

    4, Bella 2, Maria 1, Juan 3, Mark 2, Maria 4, Bella App-1 Data App-2 Data App-3 Data
  17. @karesti | @gunnarmorling #Infinispan #Debezium Distributed 1, Juan 3, Mark

    4, Bella 2, Maria 1, Juan 3, Mark 2, Maria 3, Bella App-1 Data App-2 Data App-3 Data App-4 Data
  18. @karesti | @gunnarmorling #Infinispan #Debezium Consistent Hashing N1 N2 N3

    N4
  19. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 2, Maria 4,

    Mark 2, Maria 3, Bella 4, Mark 1, Juan 3, Bella App-3 Data App-2 Data App-4 Data CH App-1 Data
  20. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 4, Mark 2,

    Maria 3, Bella 1, Juan 3, Bella 2, Maria 4, Mark App-3 Data App-2 Data App-4 Data CH App-1 Data
  21. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 4, Mark 2,

    Maria 3, Bella 1, Juan 3, Bella 2, Maria 4, Mark App-3 Data App-2 Data App-4 Data App-1 Data
  22. @karesti | @gunnarmorling #Infinispan #Debezium

  23. @karesti | @gunnarmorling #Infinispan #Debezium Microservice 1 Microservice 2 Microservice

    3 Stateless App 1 Data
  24. @karesti | @gunnarmorling #Infinispan #Debezium Infinispan Client/Server Microservice 1 Microservice

    2 hotrod 💖 Rest ... Microservice 3 Data Infinispan Server Cluster Data Data
  25. @karesti | @gunnarmorling #Infinispan #Debezium Latency chain Microservice 1 Microservice

    2 Microservice 3 Microservice 2 Distributed cache with single owner Data
  26. @karesti | @gunnarmorling #Infinispan #Debezium Data Center 1 Data Center

    2 Load Balancer APP APP Microservice APP APP Microservice Shared State Shared State Shared State Shared State Customers US Customers EU Data Data
  27. @karesti | @gunnarmorling #Infinispan #Debezium Cross Site Replication NYC +Data

    Data CEN Data LON Data RELAY2 RELAY2 RELAY2
  28. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

  29. @karesti | @gunnarmorling #Infinispan #Debezium A Common Problem: Updating Multiple

    Resources
  30. @karesti | @gunnarmorling #Infinispan #Debezium “Friends don’t let friends do

    dual-writes!” A Common Problem: Updating Multiple Resources
  31. @karesti | @gunnarmorling #Infinispan #Debezium A Better Solution Streaming Changes

    out of the Database
  32. @karesti | @gunnarmorling #Infinispan #Debezium A Better Solution Streaming Changes

    out of the Database
  33. @karesti | @gunnarmorling #Infinispan #Debezium A Better Solution Streaming Changes

    out of the Database
  34. @karesti | @gunnarmorling #Infinispan #Debezium • CDC for multiple databases

    ◦ Based on transaction logs ◦ Snapshotting, Filtering etc. • Fully open-source, very active community • Latest version: 1.4 • Production deployments at multiple companies (e.g. WePay, JW Player, Convoy, Trivago, OYO, BlaBlaCar etc.) Debezium Streaming Changes out of the Database
  35. @karesti | @gunnarmorling #Infinispan #Debezium • Stable ◦ MySQL ◦

    Postgres ◦ SQL Server ◦ MongoDB • Incubating ◦ Vitess ◦ Cassandra ◦ DB2 ◦ Oracle Debezium Supported Databases
  36. @karesti | @gunnarmorling #Infinispan #Debezium Query-based Log-based All data changes

    are captured - No polling delay or overhead - Transparent to writing applications and models - Can capture deletes and old record state - Simple Installation/Configuration - Debezium Log vs. Query-based CDC
  37. @karesti | @gunnarmorling #Infinispan #Debezium Change Data Capture Low-latency Data

    Streaming Pipelines Source: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake
  38. @karesti | @gunnarmorling #Infinispan #Debezium Change Data Capture Liberation for

    Your Data
  39. @karesti | @gunnarmorling #Infinispan #Debezium ▪ Propagate data between different

    services without coupling ▪ Each service keeps optimised views locally Microservices Data Exchange
  40. @karesti | @gunnarmorling #Infinispan #Debezium http://bit.ly/debezium-outbox-pattern Outbox Pattern

  41. @karesti | @gunnarmorling #Infinispan #Debezium Photo: “Strangler vines on trees,

    seen on the Mount Sorrow hike” by cynren, under CC BY SA 2.0 Mono-to-Micro: Strangler Fig Pattern
  42. @karesti | @gunnarmorling #Infinispan #Debezium Customer Mono-to-Micro: Strangler Fig Pattern

  43. @karesti | @gunnarmorling #Infinispan #Debezium Mono-to-Micro: Strangler Fig Pattern Customer’

    Router CDC Transformation Reads / Writes Reads Customer
  44. @karesti | @gunnarmorling #Infinispan #Debezium Customer Router CDC Reads /

    Writes Reads / Writes CDC Mono-to-Micro: Strangler Fig Pattern
  45. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka

    Streams
  46. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka

    Streams
  47. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka

    Streams
  48. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka

    Streams
  49. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka

    Streams
  50. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka

    Streams
  51. @karesti | @gunnarmorling #Infinispan #Debezium http://bit.ly/debezium-auditlogs Auditing with CDC and

    Kafka Streams
  52. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

  53. None
  54. @karesti | @gunnarmorling #Infinispan #Debezium • Fast start-up, low memory

    consumption • Developer joy • Imperative and Reactive • Best-of-breed libraries • Run via HotSpot and GraalVM native binaries Quarkus - Supersonic subatomic Java A Stack for building Cloud-native Apps
  55. @karesti | @gunnarmorling #Infinispan #Debezium • Java API for stateful

    stream processing • Rich set of operators • Scaling out to multiple JVMs • Interactive queries Kafka Streams Streaming Queries on Kafka Topics
  56. @karesti | @gunnarmorling #Infinispan #Debezium Data marshalling

  57. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

  58. @karesti | @gunnarmorling #Infinispan #Debezium Infinispan protostream

  59. @karesti | @gunnarmorling #Infinispan #Debezium ...Multi-site application with shared system-of-record

    database ✅ ...With local, denormalized read views (CQRS) ✅ ... Automatically kept in sync after writes ✅ Today’s Mission Explore how to build a... 🤩
  60. @karesti | @gunnarmorling #Infinispan #Debezium • Partial Updates, Multimap support

    • Debezium embedded into Infinispan • Kafka Connect sink connector for Infinispan ◦ https://github.com/infinispan/infinispan-kafka • CDC connector for Infinispan? Outlook
  61. @karesti | @gunnarmorling #Infinispan #Debezium • Infinispan: @infinispan | https://infinispan.org/

    • Debezium: @debezium | https://debezium.io/ • Demo: https://github.com/debezium/debezium-examples/ → distributed-caching Learn More
  62. @karesti | @gunnarmorling #Infinispan #Debezium Q & A Learn More