Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Don’t Fear Outdated Caches - Change Data Capture to the Rescue

Don’t Fear Outdated Caches - Change Data Capture to the Rescue

Joint presentation with Katia Aresti at Bordeaux JUG

The saying goes that there are only two hard things in Computer Science: cache invalidation and naming things.

Well, turns out the first is solved actually ;)

Come and join us for a session to hear not about one, but two popular open-source projects.

First, Infinispan, an in-memory distributed data store and cache, which is elastically scalable, highly availability and fault-tolerant.

Then, Debezium, a change data capture (CDC) platform for a variety of databases, such as MySQL, PostgreSQL and MongoDB.

After an overview of the key use cases and capabilities of the two projects, we'll explore in a demo how you can combine Debezium and Infinispan for even greater awesomeness:

* How to keep your cache automatically up-to-date after data changes via Debezium and CDC
* How to read denormalized views of your data with plain key look-ups from an Infinispan cluster close by
* How to propagate cache updates between clusters using cross-site replication

All this driven by Quarkus and connected via Apache Kafka. Bring your questions and ideas so we can discuss and exchange about the potential of these open-source projects in the data space!

Gunnar Morling

January 21, 2021
Tweet

More Decks by Gunnar Morling

Other Decks in Programming

Transcript

  1. Don’t Fear Outdated Caches -
    Change Data Capture to the Rescue
    Discovering Infinispan and Debezium
    Gunnar Morling
    Debezium
    Katia Aresti
    Infinispan
    January 2021 - Home Edition
    Image © Nathalie https://flic.kr/p/21Ghf2g (CC BY 2.0)

    View full-size slide

  2. @karesti | @gunnarmorling #Infinispan #Debezium
    ...Multi-site application with shared system-of-record database
    ...With local, denormalized read views (CQRS)
    ... Automatically kept in sync after writes
    Today’s Mission
    Explore how to build a...
    🤔

    View full-size slide

  3. @karesti | @gunnarmorling #Infinispan #Debezium
    Who are we?
    Gunnar Morling Katia Aresti

    View full-size slide

  4. @karesti | @gunnarmorling #Infinispan #Debezium
    Agenda
    Demo!
    Learn Learn

    View full-size slide

  5. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

    View full-size slide

  6. @karesti | @gunnarmorling #Infinispan #Debezium
    App
    Data
    Local
    In memory
    Cache Embedding a library

    View full-size slide

  7. @karesti | @gunnarmorling #Infinispan #Debezium
    App
    Data
    Local
    In memory
    Infinispan Local cache

    View full-size slide

  8. @karesti | @gunnarmorling #Infinispan #Debezium
    1, Juan
    2, Maria
    1, Juan
    App-1
    Data
    App-2
    Data
    App-3
    Data
    Local Local
    Local
    PUT
    2,Maria
    GET 2
    REMOVE 1
    Null

    View full-size slide

  9. @karesti | @gunnarmorling #Infinispan #Debezium
    1, Juan
    2, Maria
    1, Juan
    2, Maria
    1, Juan
    2, Maria
    App-1
    Data
    App-2
    Data
    App-3
    Data
    Replicated

    View full-size slide

  10. @karesti | @gunnarmorling #Infinispan #Debezium
    Potter-3
    Spells
    PUT
    3,Mark 3, Mark
    3, Mark
    Replicated
    1, Juan
    2, Maria
    1, Juan
    2, Maria
    1, Juan
    2, Maria
    App-1
    Data
    App-2
    Data
    App-3
    Data

    View full-size slide

  11. @karesti | @gunnarmorling #Infinispan #Debezium
    Replicated
    1, Juan
    2, Maria
    3, Mark
    1, Juan
    2, Maria
    3, Mark
    1, Juan
    2, Maria
    3, Mark
    App-1
    Data
    App-2
    Data
    App-3
    Data

    View full-size slide

  12. @karesti | @gunnarmorling #Infinispan #Debezium
    Distributed
    1, Juan 2, Maria
    3, Mark
    App-1
    Data
    App-2
    Data
    App-3
    Data

    View full-size slide

  13. @karesti | @gunnarmorling #Infinispan #Debezium
    GET 3
    Mark
    GET 3
    Mark
    Distributed
    1, Juan 2, Maria
    3, Mark
    App-1
    Data
    App-2
    Data
    App-3
    Data

    View full-size slide

  14. @karesti | @gunnarmorling #Infinispan #Debezium
    3, Reducto
    App-3
    Data
    Distributed
    Null
    GET 3
    1, Juan 2, Maria
    3, Mark
    App-1
    Data
    App-2
    Data

    View full-size slide

  15. @karesti | @gunnarmorling #Infinispan #Debezium
    PUT
    4, Bella
    4, Bella
    Distributed
    1, Juan
    3, Mark
    2, Maria
    1, Juan
    3, Mark
    2, Maria
    App-1
    Data
    App-2
    Data
    App-3
    Data

    View full-size slide

  16. @karesti | @gunnarmorling #Infinispan #Debezium
    Distributed
    1, Juan
    3, Mark
    4, Bella
    2, Maria
    1, Juan
    3, Mark
    2, Maria
    4, Bella
    App-1
    Data
    App-2
    Data
    App-3
    Data

    View full-size slide

  17. @karesti | @gunnarmorling #Infinispan #Debezium
    Distributed
    1, Juan
    3, Mark
    4, Bella
    2, Maria
    1, Juan
    3, Mark
    2, Maria
    3, Bella
    App-1
    Data
    App-2
    Data
    App-3
    Data
    App-4
    Data

    View full-size slide

  18. @karesti | @gunnarmorling #Infinispan #Debezium
    Consistent Hashing
    N1 N2
    N3 N4

    View full-size slide

  19. @karesti | @gunnarmorling #Infinispan #Debezium
    1, Juan
    2, Maria
    4, Mark
    2, Maria
    3, Bella
    4, Mark
    1, Juan
    3, Bella
    App-3
    Data
    App-2
    Data
    App-4
    Data
    CH
    App-1
    Data

    View full-size slide

  20. @karesti | @gunnarmorling #Infinispan #Debezium
    1, Juan
    4, Mark
    2, Maria
    3, Bella
    1, Juan
    3, Bella
    2, Maria
    4, Mark
    App-3
    Data
    App-2
    Data
    App-4
    Data
    CH
    App-1
    Data

    View full-size slide

  21. @karesti | @gunnarmorling #Infinispan #Debezium
    1, Juan
    4, Mark
    2, Maria
    3, Bella
    1, Juan
    3, Bella
    2, Maria
    4, Mark
    App-3
    Data
    App-2
    Data
    App-4
    Data
    App-1
    Data

    View full-size slide

  22. @karesti | @gunnarmorling #Infinispan #Debezium

    View full-size slide

  23. @karesti | @gunnarmorling #Infinispan #Debezium
    Microservice 1
    Microservice 2
    Microservice 3
    Stateless App 1
    Data

    View full-size slide

  24. @karesti | @gunnarmorling #Infinispan #Debezium
    Infinispan Client/Server
    Microservice 1
    Microservice 2
    hotrod 💖
    Rest
    ...
    Microservice 3
    Data
    Infinispan Server
    Cluster
    Data
    Data

    View full-size slide

  25. @karesti | @gunnarmorling #Infinispan #Debezium
    Latency chain
    Microservice 1 Microservice 2 Microservice 3
    Microservice 2
    Distributed cache
    with single owner
    Data

    View full-size slide

  26. @karesti | @gunnarmorling #Infinispan #Debezium
    Data Center 1
    Data Center 2
    Load Balancer
    APP
    APP
    Microservice
    APP
    APP
    Microservice
    Shared
    State
    Shared
    State
    Shared
    State
    Shared
    State
    Customers US
    Customers EU
    Data
    Data

    View full-size slide

  27. @karesti | @gunnarmorling #Infinispan #Debezium
    Cross Site Replication
    NYC
    +Data
    Data
    CEN
    Data
    LON
    Data
    RELAY2
    RELAY2
    RELAY2

    View full-size slide

  28. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

    View full-size slide

  29. @karesti | @gunnarmorling #Infinispan #Debezium
    A Common Problem: Updating Multiple Resources

    View full-size slide

  30. @karesti | @gunnarmorling #Infinispan #Debezium
    “Friends don’t let friends
    do dual-writes!”
    A Common Problem: Updating Multiple Resources

    View full-size slide

  31. @karesti | @gunnarmorling #Infinispan #Debezium
    A Better Solution
    Streaming Changes out of the Database

    View full-size slide

  32. @karesti | @gunnarmorling #Infinispan #Debezium
    A Better Solution
    Streaming Changes out of the Database

    View full-size slide

  33. @karesti | @gunnarmorling #Infinispan #Debezium
    A Better Solution
    Streaming Changes out of the Database

    View full-size slide

  34. @karesti | @gunnarmorling #Infinispan #Debezium
    ● CDC for multiple databases
    ○ Based on transaction logs
    ○ Snapshotting, Filtering etc.
    ● Fully open-source, very active community
    ● Latest version: 1.4
    ● Production deployments at multiple companies
    (e.g. WePay, JW Player, Convoy, Trivago, OYO,
    BlaBlaCar etc.)
    Debezium
    Streaming Changes out of the Database

    View full-size slide

  35. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Stable
    ○ MySQL
    ○ Postgres
    ○ SQL Server
    ○ MongoDB
    ● Incubating
    ○ Vitess
    ○ Cassandra
    ○ DB2
    ○ Oracle
    Debezium
    Supported Databases

    View full-size slide

  36. @karesti | @gunnarmorling #Infinispan #Debezium
    Query-based Log-based
    All data changes are captured -
    No polling delay or overhead -
    Transparent to writing applications
    and models
    -
    Can capture deletes and old record
    state
    -
    Simple Installation/Configuration -
    Debezium
    Log vs. Query-based CDC

    View full-size slide

  37. @karesti | @gunnarmorling #Infinispan #Debezium
    Change Data Capture
    Low-latency Data Streaming Pipelines
    Source: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake

    View full-size slide

  38. @karesti | @gunnarmorling #Infinispan #Debezium
    Change Data Capture
    Liberation for Your Data

    View full-size slide

  39. @karesti | @gunnarmorling #Infinispan #Debezium
    ■ Propagate data between different
    services without coupling
    ■ Each service keeps
    optimised views locally
    Microservices Data Exchange

    View full-size slide

  40. @karesti | @gunnarmorling #Infinispan #Debezium
    http://bit.ly/debezium-outbox-pattern
    Outbox Pattern

    View full-size slide

  41. @karesti | @gunnarmorling #Infinispan #Debezium
    Photo: “Strangler vines on trees, seen on the Mount Sorrow hike” by cynren, under CC BY SA 2.0
    Mono-to-Micro: Strangler Fig Pattern

    View full-size slide

  42. @karesti | @gunnarmorling #Infinispan #Debezium
    Customer
    Mono-to-Micro: Strangler Fig Pattern

    View full-size slide

  43. @karesti | @gunnarmorling #Infinispan #Debezium
    Mono-to-Micro: Strangler Fig Pattern
    Customer’
    Router
    CDC
    Transformation
    Reads /
    Writes Reads
    Customer

    View full-size slide

  44. @karesti | @gunnarmorling #Infinispan #Debezium
    Customer
    Router
    CDC
    Reads /
    Writes
    Reads /
    Writes
    CDC
    Mono-to-Micro: Strangler Fig Pattern

    View full-size slide

  45. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams

    View full-size slide

  46. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams

    View full-size slide

  47. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams

    View full-size slide

  48. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams

    View full-size slide

  49. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams

    View full-size slide

  50. @karesti | @gunnarmorling #Infinispan #Debezium
    Auditing with CDC and Kafka Streams

    View full-size slide

  51. @karesti | @gunnarmorling #Infinispan #Debezium
    http://bit.ly/debezium-auditlogs
    Auditing with CDC and Kafka Streams

    View full-size slide

  52. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
    Demo

    View full-size slide

  53. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Fast start-up, low memory consumption
    ● Developer joy
    ● Imperative and Reactive
    ● Best-of-breed libraries
    ● Run via HotSpot and GraalVM native binaries
    Quarkus - Supersonic subatomic Java
    A Stack for building Cloud-native Apps

    View full-size slide

  54. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Java API for stateful stream processing
    ● Rich set of operators
    ● Scaling out to multiple JVMs
    ● Interactive queries
    Kafka Streams
    Streaming Queries on Kafka Topics

    View full-size slide

  55. @karesti | @gunnarmorling #Infinispan #Debezium
    Data marshalling

    View full-size slide

  56. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
    Demo

    View full-size slide

  57. @karesti | @gunnarmorling #Infinispan #Debezium
    Infinispan protostream

    View full-size slide

  58. @karesti | @gunnarmorling #Infinispan #Debezium
    ...Multi-site application with shared system-of-record database ✅
    ...With local, denormalized read views (CQRS) ✅
    ... Automatically kept in sync after writes ✅
    Today’s Mission
    Explore how to build a...
    🤩

    View full-size slide

  59. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Partial Updates, Multimap support
    ● Debezium embedded into Infinispan
    ● Kafka Connect sink connector for Infinispan
    ○ https://github.com/infinispan/infinispan-kafka
    ● CDC connector for Infinispan?
    Outlook

    View full-size slide

  60. @karesti | @gunnarmorling #Infinispan #Debezium
    ● Infinispan: @infinispan | https://infinispan.org/
    ● Debezium: @debezium | https://debezium.io/
    ● Demo: https://github.com/debezium/debezium-examples/ →
    distributed-caching
    Learn More

    View full-size slide

  61. @karesti | @gunnarmorling #Infinispan #Debezium
    Q & A
    Learn More

    View full-size slide