Don’t Fear Outdated Caches - Change Data Capture to the Rescue

Joint presentation with Katia Aresti at Bordeaux JUG

The saying goes that there are only two hard things in Computer Science: cache invalidation and naming things.

Well, turns out the first is solved actually ;)

Come and join us for a session to hear not about one, but two popular open-source projects.

First, Infinispan, an in-memory distributed data store and cache, which is elastically scalable, highly availability and fault-tolerant.

Then, Debezium, a change data capture (CDC) platform for a variety of databases, such as MySQL, PostgreSQL and MongoDB.

After an overview of the key use cases and capabilities of the two projects, we'll explore in a demo how you can combine Debezium and Infinispan for even greater awesomeness:

* How to keep your cache automatically up-to-date after data changes via Debezium and CDC
* How to read denormalized views of your data with plain key look-ups from an Infinispan cluster close by
* How to propagate cache updates between clusters using cross-site replication

All this driven by Quarkus and connected via Apache Kafka. Bring your questions and ideas so we can discuss and exchange about the potential of these open-source projects in the data space!

Transcript

1. Don’t Fear Outdated Caches -
   Change Data Capture to the Rescue
   Discovering Infinispan and Debezium
   Gunnar Morling
   Debezium
   Katia Aresti
   Infinispan
   January 2021 - Home Edition
   Image © Nathalie https://flic.kr/p/21Ghf2g (CC BY 2.0)
   

   View Slide

2. @karesti | @gunnarmorling #Infinispan #Debezium
   ...Multi-site application with shared system-of-record database
   ...With local, denormalized read views (CQRS)
   ... Automatically kept in sync after writes
   Today’s Mission
   Explore how to build a...
   🤔
   

   View Slide

3. @karesti | @gunnarmorling #Infinispan #Debezium
   Who are we?
   Gunnar Morling Katia Aresti
   

   View Slide

4. @karesti | @gunnarmorling #Infinispan #Debezium
   Agenda
   Demo!
   Learn Learn
   

   View Slide

5. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
   

   View Slide

6. @karesti | @gunnarmorling #Infinispan #Debezium
   App
   Data
   Local
   In memory
   Cache Embedding a library
   

   View Slide

7. @karesti | @gunnarmorling #Infinispan #Debezium
   App
   Data
   Local
   In memory
   Infinispan Local cache
   

   View Slide

8. @karesti | @gunnarmorling #Infinispan #Debezium
   1, Juan
   2, Maria
   1, Juan
   App-1
   Data
   App-2
   Data
   App-3
   Data
   Local Local
   Local
   PUT
   2,Maria
   GET 2
   REMOVE 1
   Null
   

   View Slide

9. @karesti | @gunnarmorling #Infinispan #Debezium
   1, Juan
   2, Maria
   1, Juan
   2, Maria
   1, Juan
   2, Maria
   App-1
   Data
   App-2
   Data
   App-3
   Data
   Replicated
   

   View Slide

10. @karesti | @gunnarmorling #Infinispan #Debezium
    Potter-3
    Spells
    PUT
    3,Mark 3, Mark
    3, Mark
    Replicated
    1, Juan
    2, Maria
    1, Juan
    2, Maria
    1, Juan
    2, Maria
    App-1
    Data
    App-2
    Data
    App-3
    Data
    

    View Slide

11. @karesti | @gunnarmorling #Infinispan #Debezium
    Replicated
    1, Juan
    2, Maria
    3, Mark
    1, Juan
    2, Maria
    3, Mark
    1, Juan
    2, Maria
    3, Mark
    App-1
    Data
    App-2
    Data
    App-3
    Data
    

    View Slide

12. @karesti | @gunnarmorling #Infinispan #Debezium
    Distributed
    1, Juan 2, Maria
    3, Mark
    App-1
    Data
    App-2
    Data
    App-3
    Data
    

    View Slide

13. @karesti | @gunnarmorling #Infinispan #Debezium
    GET 3
    Mark
    GET 3
    Mark
    Distributed
    1, Juan 2, Maria
    3, Mark
    App-1
    Data
    App-2
    Data
    App-3
    Data
    

    View Slide

14. @karesti | @gunnarmorling #Infinispan #Debezium
    3, Reducto
    App-3
    Data
    Distributed
    Null
    GET 3
    1, Juan 2, Maria
    3, Mark
    App-1
    Data
    App-2
    Data
    

    View Slide

15. @karesti | @gunnarmorling #Infinispan #Debezium
    PUT
    4, Bella
    4, Bella
    Distributed
    1, Juan
    3, Mark
    2, Maria
    1, Juan
    3, Mark
    2, Maria
    App-1
    Data
    App-2
    Data
    App-3
    Data
    

    View Slide

16. @karesti | @gunnarmorling #Infinispan #Debezium
    Distributed
    1, Juan
    3, Mark
    4, Bella
    2, Maria
    1, Juan
    3, Mark
    2, Maria
    4, Bella
    App-1
    Data
    App-2
    Data
    App-3
    Data
    

    View Slide

17. @karesti | @gunnarmorling #Infinispan #Debezium
    Distributed
    1, Juan
    3, Mark
    4, Bella
    2, Maria
    1, Juan
    3, Mark
    2, Maria
    3, Bella
    App-1
    Data
    App-2
    Data
    App-3
    Data
    App-4
    Data
    

    View Slide

18. @karesti | @gunnarmorling #Infinispan #Debezium
    Consistent Hashing
    N1 N2
    N3 N4
    

    View Slide

19. @karesti | @gunnarmorling #Infinispan #Debezium
    1, Juan
    2, Maria
    4, Mark
    2, Maria
    3, Bella
    4, Mark
    1, Juan
    3, Bella
    App-3
    Data
    App-2
    Data
    App-4
    Data
    CH
    App-1
    Data
    

    View Slide

20. @karesti | @gunnarmorling #Infinispan #Debezium
    1, Juan
    4, Mark
    2, Maria
    3, Bella
    1, Juan
    3, Bella
    2, Maria
    4, Mark
    App-3
    Data
    App-2
    Data
    App-4
    Data
    CH
    App-1
    Data
    

    View Slide

21. @karesti | @gunnarmorling #Infinispan #Debezium
    1, Juan
    4, Mark
    2, Maria
    3, Bella
    1, Juan
    3, Bella
    2, Maria
    4, Mark
    App-3
    Data
    App-2
    Data
    App-4
    Data
    App-1
    Data
    

    View Slide

22. @karesti | @gunnarmorling #Infinispan #Debezium
    

    View Slide

23. @karesti | @gunnarmorling #Infinispan #Debezium
    Microservice 1
    Microservice 2
    Microservice 3
    Stateless App 1
    Data
    

    View Slide

24. @karesti | @gunnarmorling #Infinispan #Debezium
    Infinispan Client/Server
    Microservice 1
    Microservice 2
    hotrod 💖
    Rest
    ...
    Microservice 3
    Data
    Infinispan Server
    Cluster
    Data
    Data
    

    View Slide

25. @karesti | @gunnarmorling #Infinispan #Debezium
    Latency chain
    Microservice 1 Microservice 2 Microservice 3
    Microservice 2
    Distributed cache
    with single owner
    Data
    

    View Slide

26. @karesti | @gunnarmorling #Infinispan #Debezium
    Data Center 1
    Data Center 2
    Load Balancer
    APP
    APP
    Microservice
    APP
    APP
    Microservice
    Shared
    State
    Shared
    State
    Shared
    State
    Shared
    State
    Customers US
    Customers EU
    Data
    Data
    

    View Slide

27. @karesti | @gunnarmorling #Infinispan #Debezium
    Cross Site Replication
    NYC
    +Data
    Data
    CEN
    Data
    LON
    Data
    RELAY2
    RELAY2
    RELAY2
    

    View Slide

28. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
    

    View Slide

29. @karesti | @gunnarmorling #Infinispan #Debezium
    A Common Problem: Updating Multiple Resources
    

    View Slide

30. @karesti | @gunnarmorling #Infinispan #Debezium
    “Friends don’t let friends
    do dual-writes!”
    A Common Problem: Updating Multiple Resources
    

    View Slide

31. @karesti | @gunnarmorling #Infinispan #Debezium
    A Better Solution
    Streaming Changes out of the Database
    

    View Slide

32. @karesti | @gunnarmorling #Infinispan #Debezium
    A Better Solution
    Streaming Changes out of the Database
    

    View Slide

33. @karesti | @gunnarmorling #Infinispan #Debezium
    A Better Solution
    Streaming Changes out of the Database
    

    View Slide

34. @karesti | @gunnarmorling #Infinispan #Debezium
    ● CDC for multiple databases
    ○ Based on transaction logs
    ○ Snapshotting, Filtering etc.
    ● Fully open-source, very active community
    ● Latest version: 1.4
    ● Production deployments at multiple companies
    (e.g. WePay, JW Player, Convoy, Trivago, OYO,
    BlaBlaCar etc.)
    Debezium
    Streaming Changes out of the Database
    

    View Slide

35. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Stable
    ○ MySQL
    ○ Postgres
    ○ SQL Server
    ○ MongoDB
    ● Incubating
    ○ Vitess
    ○ Cassandra
    ○ DB2
    ○ Oracle
    Debezium
    Supported Databases
    

    View Slide

36. @karesti | @gunnarmorling #Infinispan #Debezium
    Query-based Log-based
    All data changes are captured -
    No polling delay or overhead -
    Transparent to writing applications
    and models
    -
    Can capture deletes and old record
    state
    -
    Simple Installation/Configuration -
    Debezium
    Log vs. Query-based CDC
    

    View Slide

37. @karesti | @gunnarmorling #Infinispan #Debezium
    Change Data Capture
    Low-latency Data Streaming Pipelines
    Source: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake
    

    View Slide

38. @karesti | @gunnarmorling #Infinispan #Debezium
    Change Data Capture
    Liberation for Your Data
    

    View Slide

39. @karesti | @gunnarmorling #Infinispan #Debezium
    ■ Propagate data between different
    services without coupling
    ■ Each service keeps
    optimised views locally
    Microservices Data Exchange
    

    View Slide

40. @karesti | @gunnarmorling #Infinispan #Debezium
    http://bit.ly/debezium-outbox-pattern
    Outbox Pattern
    

    View Slide

41. @karesti | @gunnarmorling #Infinispan #Debezium
    Photo: “Strangler vines on trees, seen on the Mount Sorrow hike” by cynren, under CC BY SA 2.0
    Mono-to-Micro: Strangler Fig Pattern
    

    View Slide

42. @karesti | @gunnarmorling #Infinispan #Debezium
    Customer
    Mono-to-Micro: Strangler Fig Pattern
    

    View Slide

43. @karesti | @gunnarmorling #Infinispan #Debezium
    Mono-to-Micro: Strangler Fig Pattern
    Customer’
    Router
    CDC
    Transformation
    Reads /
    Writes Reads
    Customer
    

    View Slide

44. @karesti | @gunnarmorling #Infinispan #Debezium
    Customer
    Router
    CDC
    Reads /
    Writes
    Reads /
    Writes
    CDC
    Mono-to-Micro: Strangler Fig Pattern
    

    View Slide

45. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams
    

    View Slide

46. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams
    

    View Slide

47. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams
    

    View Slide

48. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams
    

    View Slide

49. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams
    

    View Slide

50. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams
    

    View Slide

51. @karesti | @gunnarmorling #Infinispan #Debezium
    http://bit.ly/debezium-auditlogs
    Auditing with CDC and Kafka Streams
    

    View Slide

52. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
    Demo
    

    View Slide

53. View Slide

54. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Fast start-up, low memory consumption
    ● Developer joy
    ● Imperative and Reactive
    ● Best-of-breed libraries
    ● Run via HotSpot and GraalVM native binaries
    Quarkus - Supersonic subatomic Java
    A Stack for building Cloud-native Apps
    

    View Slide

55. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Java API for stateful stream processing
    ● Rich set of operators
    ● Scaling out to multiple JVMs
    ● Interactive queries
    Kafka Streams
    Streaming Queries on Kafka Topics
    

    View Slide

56. @karesti | @gunnarmorling #Infinispan #Debezium
    Data marshalling
    

    View Slide

57. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
    Demo
    

    View Slide

58. @karesti | @gunnarmorling #Infinispan #Debezium
    Infinispan protostream
    

    View Slide

59. @karesti | @gunnarmorling #Infinispan #Debezium
    ...Multi-site application with shared system-of-record database ✅
    ...With local, denormalized read views (CQRS) ✅
    ... Automatically kept in sync after writes ✅
    Today’s Mission
    Explore how to build a...
    🤩
    

    View Slide

60. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Partial Updates, Multimap support
    ● Debezium embedded into Infinispan
    ● Kafka Connect sink connector for Infinispan
    ○ https://github.com/infinispan/infinispan-kafka
    ● CDC connector for Infinispan?
    Outlook
    

    View Slide

61. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Infinispan: @infinispan | https://infinispan.org/
    ● Debezium: @debezium | https://debezium.io/
    ● Demo: https://github.com/debezium/debezium-examples/ →
    distributed-caching
    Learn More
    

    View Slide

62. @karesti | @gunnarmorling #Infinispan #Debezium
    Q & A
    Learn More
    

    View Slide