$30 off During Our Annual Pro Sale. View Details »

Keep your cache always fresh with Debezium! (Current 22)

Keep your cache always fresh with Debezium! (Current 22)

The saying goes that there are only two hard things in Computer Science: cache invalidation, and naming things. Well, turns out the first one is solved actually. Join us for this session to learn how to keep read views of your data in distributed caches close to your users, always kept in sync with your primary data stores change data capture. You will learn how to:

* Implement a low-latency data pipeline for cache updates based on Debezium, Apache Kafka, and Infinispan
* Create denormalized views of your data using Kafka Streams and make them accessible via plain key look-ups from a cache cluster close by
* Propagate updates between cache clusters using cross-site replication

We'll also touch on some advanced concepts, such as detecting and rejecting writes to the system of record which are derived from outdated cached state, and show in a demo how all the pieces come together, of course connected via Apache Kafka.

Gunnar Morling

October 05, 2022
Tweet

More Decks by Gunnar Morling

Other Decks in Programming

Transcript

  1. Image © Nathalie https://flic.kr/p/21Ghf2g (CC BY 2.0) Keep Your Cache

    Always Fresh With Debezium! Gunnar Morling Software Engineer, Red Hat @gunnarmorling
  2. #Debezium @gunnarmorling The Challenge Multi-site Application With Shared Database

  3. #Debezium @gunnarmorling The Challenge Multi-site Application With Shared Database

  4. #Debezium @gunnarmorling The Challenge Multi-site Application With Shared Database

  5. #Debezium @gunnarmorling The Challenge • Deployments in different geographies, single

    DB • 90% read requests • Complex queries Multi-site Application With Shared Database
  6. #Debezium @gunnarmorling … Multi-site application with shared system-of-record database …

    With local, denormalized read views (CQRS) ... Automatically kept in sync after writes Today’s Mission 🤔 Explore How to Build a…
  7. #Debezium @gunnarmorling • Software engineer at Red Hat ◦ Debezium

    ◦ Quarkus • kcctl 🧸, JfrUnit, ModiTect, MapStruct • Spec Lead for Bean Validation 2.0 • Java Champion Gunnar Morling
  8. #Debezium @gunnarmorling The Idea Caching to the Rescue!

  9. #Debezium @gunnarmorling The Idea Caching to the Rescue!

  10. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

  11. #Debezium @gunnarmorling Infinispan 100% Open-source In-Memory Distributed Data Store Interoperability

    Resilient Fault Tolerant Data Clustered Processing Query ACID Tx
  12. #Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria

    2, Jenny 2, Jenny 3, Juan App 3 Data 1, Maria 3, Juan Infinispan Deployment Distributed Cache
  13. #Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria

    2, Jenny 2, Jenny 3, Juan App 3 Data 1, Maria 3, Juan Put 4 Will Put 4 Will Infinispan Deployment Distributed Cache
  14. #Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria

    2, Jenny 4, Will 2, Jenny 3, Juan 4, Will App 3 Data 1, Maria 3, Juan Infinispan Deployment Distributed Cache
  15. #Debezium @gunnarmorling

  16. #Debezium @gunnarmorling Infinispan Client/Server Service 1 Service 2 Binary (hot

    rod) REST ... Service 3 Data Infinispan Cluster Data Data
  17. #Debezium @gunnarmorling Infinispan Cross-Site Replication AWS (LON) GCP (NYC) Load

    Balancer APP APP Service APP APP Service Shared State Shared State Shared State Shared State Data Data Data NYC Data LON RELAY2
  18. #Debezium @gunnarmorling The Question How To Keep The Cache In

    Sync?
  19. #Debezium @gunnarmorling Dual Writes Easy to Get Wrong

  20. https://flic.kr/p/PFDvkY Public Domain, Angelo Br

  21. #Debezium @gunnarmorling Debezium in a Nutshell Open-Source Change Data Capture

    • A CDC Platform ◦ Based on transaction logs ◦ Snapshotting, filtering, etc. ◦ Outbox support ◦ Web-based UI • Fully open-source, very active community • Large production deployments
  22. #Debezium @gunnarmorling Debezium Becoming the De-Facto CDC Standard https://debezium.io/blog/2021/09/22/deep-dive-into-a-debezium-community-connector-scylla-cdc-source-connector/

  23. #Debezium @gunnarmorling Debezium: Deployment Alternatives Embedded Engine and Debezium Server

  24. #Debezium @gunnarmorling Solution Overview Capturing Changes From the Database

  25. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

  26. None
  27. #Debezium @gunnarmorling • Fast start-up, low memory consumption • Developer

    joy • Imperative and Reactive • Best-of-breed libraries • Run via HotSpot and GraalVM native binaries Quarkus - Supersonic Subatomic Java A Stack for Building Cloud-native Apps
  28. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

  29. #Debezium @gunnarmorling Eventual Consistency Dealing With Stale Data • Cached

    views updated asynchronously • How to ensure read your own writes semantics? • How to prevent taking action based on stale data? 🤔
  30. #Debezium @gunnarmorling Eventual Consistency Read Your Own Writes

  31. #Debezium @gunnarmorling Eventual Consistency Read Your Own Writes

  32. #Debezium @gunnarmorling Eventual Consistency Read Your Own Writes

  33. #Debezium @gunnarmorling Eventual Consistency Read Your Own Writes

  34. #Debezium @gunnarmorling Eventual Consistency Detecting Writes Derived From Stale State

  35. Wrap-Up

  36. #Debezium @gunnarmorling … Multi-site application with shared system-of-record database ✅

    … With local, denormalized read views (CQRS) ✅ ... Automatically kept in sync after writes ✅ Today’s Mission 🤩 Explore How to Build a…
  37. #Debezium @gunnarmorling Discussion What Have We Gained? • Pro: ◦

    Lower latencies ◦ Reduced load on primary database ◦ Increased availability • Con: ◦ Increased complexity
  38. #Debezium @gunnarmorling • Incremental snapshotting • Postgres logical decoding messages

    • Multi-DB support (SQL Server) • Debezium Server sinks • MongoDB change streams support • Debezium UI • Debezium 2.0 What’s New in Debezium?
  39. #Debezium @gunnarmorling • Infinispan: @infinispan | https://infinispan.org/ • Debezium: @debezium

    | https://debezium.io/ • Demo: https://github.com/debezium/debezium-examples/ → distributed-caching • kcctl 🧸: https://github.com/kcctl/kcctl/ Learn More
  40. #Debezium @gunnarmorling Q & A gunnar@hibernate.org @gunnarmorling 📧 Thank You!

  41. None