Keep Your Cache Always Fresh With Debezium

Slide 1

Slide 1 text

Image © Nathalie https://flic.kr/p/21Ghf2g (CC BY 2.0) Keep Your Cache Always Fresh With Debezium! Gunnar Morling Software Engineer, Red Hat @gunnarmorling

Slide 2

Slide 2 text

#Debezium @gunnarmorling The Challenge ● 90% read requests ● Complex queries Multi-site Application With Shared Database

Slide 3

Slide 3 text

#Debezium @gunnarmorling … Multi-site application with shared system-of-record database … With local, denormalized read views (CQRS) ... Automatically kept in sync after writes Today’s Mission 🤔 Explore How to Build a…

Slide 4

Slide 4 text

#Debezium @gunnarmorling Agenda Demo! Learn Learn

Slide 5

Slide 5 text

#Debezium @gunnarmorling ● Software engineer at Red Hat ○ Debezium ○ Quarkus ● kcctl 🧸, JfrUnit, ModiTect, MapStruct ● Spec Lead for Bean Validation 2.0 ● Java Champion ● @gunnarmorling Gunnar Morling

Slide 6

Slide 6 text

#Debezium @gunnarmorling The Idea Caching to the Rescue!

Slide 7

Slide 7 text

#Debezium @gunnarmorling The Idea Caching to the Rescue!

Slide 8

Slide 8 text

https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

Slide 9

Slide 9 text

#Debezium @gunnarmorling Inﬁnispan 100% Open-source In-Memory Distributed Data Store Interoperability Resilient Fault Tolerant Data Clustered Processing Query ACID Tx

Slide 10

Slide 10 text

#Debezium @gunnarmorling Inﬁnispan Deployment Local Cache App Data 5, Peter

Slide 11

Slide 11 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 5, Peter 5, Peter App 3 Data Put 4, Mike Remove 5 Peter Get 2, Null Inﬁnispan Deployment Local Cache

Slide 12

Slide 12 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 1, Maria 2, Jenny App 3 Data 1, Maria 2, Jenny Inﬁnispan Deployment Replicated Cache

Slide 13

Slide 13 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 1, Maria 2, Jenny App 3 Data 1, Maria 2, Jenny Put 3, Juan 3, Juan Inﬁnispan Deployment Replicated Cache

Slide 14

Slide 14 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 3, Juan 1, Maria 2, Jenny 3, Juan App 3 Data 1, Maria 2, Jenny 3, Juan Inﬁnispan Deployment Replicated Cache

Slide 15

Slide 15 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny App 3 Data 3, Juan Inﬁnispan Deployment Distributed Cache (One Owner)

Slide 16

Slide 16 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny App 3 Data 3, Juan Get 2 Jenny Get 2 Jenny Inﬁnispan Deployment Distributed Cache (One Owner)

Slide 17

Slide 17 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny App 3 Data 3, Juan Inﬁnispan Deployment Distributed Cache (One Owner)

Slide 18

Slide 18 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 2, Jenny 3, Juan App 3 Data 1, Maria 3, Juan Inﬁnispan Deployment Distributed Cache (Two Owners)

Slide 19

Slide 19 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 2, Jenny 3, Juan App 3 Data 1, Maria 3, Juan Put 4 Will Put 4 Will Inﬁnispan Deployment Distributed Cache (Two Owners)

Slide 20

Slide 20 text

#Debezium @gunnarmorling App 1 Data App 2 Data 1, Maria 2, Jenny 4, Will 2, Jenny 3, Juan 4, Will App 3 Data 1, Maria 3, Juan Inﬁnispan Deployment Distributed Cache (Two Owners)

Slide 21

Slide 21 text

#Debezium @gunnarmorling

Slide 22

Slide 22 text

#Debezium @gunnarmorling Inﬁnispan Client/Server Service 1 Service 2 Binary (hot rod) REST ... Service 3 Data Infinispan Cluster Data Data

Slide 23

Slide 23 text

#Debezium @gunnarmorling Inﬁnispan Cross-Site Replication AWS (LON) GCP (NYC) Load Balancer APP APP Service APP APP Service Shared State Shared State Shared State Shared State Data Data Data NYC Data LON RELAY2

Slide 24

Slide 24 text

#Debezium @gunnarmorling The Question How To Keep The Cache In Sync?

Slide 25

Slide 25 text

https://flic.kr/p/PFDvkY Public Domain, Angelo Br

Slide 26

Slide 26 text

#Debezium @gunnarmorling Debezium in a Nutshell Open-Source Change Data Capture ● A CDC Platform ○ Based on transaction logs ○ Snapshotting, filtering, etc. ○ Outbox support ○ Web-based UI ● Fully open-source, very active community ● Large production deployments

Slide 27

Slide 27 text

#Debezium @gunnarmorling Change Data Capture Liberation for Your Data

Slide 28

Slide 28 text

#Debezium @gunnarmorling Change Data Capture Liberation for Your Data

Slide 29

Slide 29 text

#Debezium @gunnarmorling Solution Overview Capturing Changes From the Database

Slide 30

Slide 30 text

#Debezium @gunnarmorling ● Community-hosted connectors ● pg_logical_emit_message() ● Multi-DB support (SQL Server) ● Debezium Server sinks ● MongoDB change streams support ● Debezium UI ● etc. Detour: What’s New in Debezium?

Slide 31

Slide 31 text

#Debezium @gunnarmorling ● Can’t update filter list ● Long-running snapshots can’t be paused/resumed ● Can’t stream changes until snapshot completed ● Can’t re-snapshot selected tables Detour: What’s New in Debezium? Incremental Snapshotting

Slide 32

Slide 32 text

#Debezium @gunnarmorling Incremental Snapshotting ● “DBLog: A Watermark Based Change-Data-Capture Framework”, by Andreas Andreakis and Ioannis Papapanagiotou ● Key idea: interleave snapshot events and events from TX log https://arxiv.org/pdf/2010.12597v1.pdf Detour: What’s New in Debezium?

Slide 33

Slide 33 text

#Debezium @gunnarmorling Incremental Snapshotting Detour: What’s New in Debezium?

Slide 34

Slide 34 text

#Debezium @gunnarmorling Incremental Snapshotting Detour: What’s New in Debezium?

Slide 35

Slide 35 text

#Debezium @gunnarmorling Support for pg_logical_emit_message() Detour: What’s New in Debezium? ● Directly writing arbitrary messages to the WAL ● No need for an outbox table

Slide 36

Slide 36 text

#Debezium @gunnarmorling Multi-DB Support Detour: What’s New in Debezium?

Slide 37

Slide 37 text

https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

#Debezium @gunnarmorling ● Fast start-up, low memory consumption ● Developer joy ● Imperative and Reactive ● Best-of-breed libraries ● Run via HotSpot and GraalVM native binaries Quarkus - Supersonic Subatomic Java A Stack for Building Cloud-native Apps

Slide 40

Slide 40 text

#Debezium @gunnarmorling ● Java API for stateful stream processing ● Rich set of operators ● Scaling out to multiple JVMs ● Interactive queries Kafka Streams Streaming Queries on Kafka Topics

Slide 41

Slide 41 text

https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

Slide 42

Slide 42 text

#Debezium @gunnarmorling … Multi-site application with shared system-of-record database ✅ … With local, denormalized read views (CQRS) ✅ ... Automatically kept in sync after writes ✅ Today’s Mission 🤩 Explore How to Build a…

Slide 43

Slide 43 text

#Debezium @gunnarmorling ● Infinispan: @infinispan | https://infinispan.org/ ● Debezium: @debezium | https://debezium.io/ ● Demo: https://github.com/debezium/debezium-examples/ → distributed-caching ● kcctl 🧸: https://github.com/kcctl/kcctl/ Learn More

Slide 44

Slide 44 text

44 Try our Kafka service! ▸ 48 hour trial ▸ Free of charge ▸ OpenShift and Kubernetes AppDev examples Red Hat OpenShift Streams for Apache Kafka Trial TRY IT TODAY! http://red.ht/TryKafka