Tweet
Tweet

Slide 1

Slide 1 text

Don’t Fear Outdated Caches - Change Data Capture to the Rescue Discovering Infinispan and Debezium Gunnar Morling Debezium Katia Aresti Infinispan January 2021 - Home Edition Image © Nathalie https://flic.kr/p/21Ghf2g (CC BY 2.0)

Slide 2

Slide 2 text

@karesti | @gunnarmorling #Infinispan #Debezium ...Multi-site application with shared system-of-record database ...With local, denormalized read views (CQRS) ... Automatically kept in sync after writes Today’s Mission Explore how to build a... 🤔

Slide 3

Slide 3 text

@karesti | @gunnarmorling #Infinispan #Debezium Who are we? Gunnar Morling Katia Aresti

Slide 4

Slide 4 text

@karesti | @gunnarmorling #Infinispan #Debezium Agenda Demo! Learn Learn

Slide 5

Slide 5 text

https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

Slide 6

Slide 6 text

@karesti | @gunnarmorling #Infinispan #Debezium App Data Local In memory Cache Embedding a library

Slide 7

Slide 7 text

@karesti | @gunnarmorling #Infinispan #Debezium App Data Local In memory Infinispan Local cache

Slide 8

Slide 8 text

@karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 2, Maria 1, Juan App-1 Data App-2 Data App-3 Data Local Local Local PUT 2,Maria GET 2 REMOVE 1 Null

Slide 9

Slide 9 text

@karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 2, Maria 1, Juan 2, Maria 1, Juan 2, Maria App-1 Data App-2 Data App-3 Data Replicated

Slide 10

Slide 10 text

@karesti | @gunnarmorling #Infinispan #Debezium Potter-3 Spells PUT 3,Mark 3, Mark 3, Mark Replicated 1, Juan 2, Maria 1, Juan 2, Maria 1, Juan 2, Maria App-1 Data App-2 Data App-3 Data

Slide 11

Slide 11 text

@karesti | @gunnarmorling #Infinispan #Debezium Replicated 1, Juan 2, Maria 3, Mark 1, Juan 2, Maria 3, Mark 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data App-3 Data

Slide 12

Slide 12 text

@karesti | @gunnarmorling #Infinispan #Debezium Distributed 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data App-3 Data

Slide 13

Slide 13 text

@karesti | @gunnarmorling #Infinispan #Debezium GET 3 Mark GET 3 Mark Distributed 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data App-3 Data

Slide 14

Slide 14 text

@karesti | @gunnarmorling #Infinispan #Debezium 3, Reducto App-3 Data Distributed Null GET 3 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data

Slide 15

Slide 15 text

@karesti | @gunnarmorling #Infinispan #Debezium PUT 4, Bella 4, Bella Distributed 1, Juan 3, Mark 2, Maria 1, Juan 3, Mark 2, Maria App-1 Data App-2 Data App-3 Data

Slide 16

Slide 16 text

@karesti | @gunnarmorling #Infinispan #Debezium Distributed 1, Juan 3, Mark 4, Bella 2, Maria 1, Juan 3, Mark 2, Maria 4, Bella App-1 Data App-2 Data App-3 Data

Slide 17

Slide 17 text

@karesti | @gunnarmorling #Infinispan #Debezium Distributed 1, Juan 3, Mark 4, Bella 2, Maria 1, Juan 3, Mark 2, Maria 3, Bella App-1 Data App-2 Data App-3 Data App-4 Data

Slide 18

Slide 18 text

@karesti | @gunnarmorling #Infinispan #Debezium Consistent Hashing N1 N2 N3 N4

Slide 19

Slide 19 text

@karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 2, Maria 4, Mark 2, Maria 3, Bella 4, Mark 1, Juan 3, Bella App-3 Data App-2 Data App-4 Data CH App-1 Data

Slide 20

Slide 20 text

@karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 4, Mark 2, Maria 3, Bella 1, Juan 3, Bella 2, Maria 4, Mark App-3 Data App-2 Data App-4 Data CH App-1 Data

Slide 21

Slide 21 text

@karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 4, Mark 2, Maria 3, Bella 1, Juan 3, Bella 2, Maria 4, Mark App-3 Data App-2 Data App-4 Data App-1 Data

Slide 22

Slide 22 text

@karesti | @gunnarmorling #Infinispan #Debezium

Slide 23

Slide 23 text

@karesti | @gunnarmorling #Infinispan #Debezium Microservice 1 Microservice 2 Microservice 3 Stateless App 1 Data

Slide 24

Slide 24 text

@karesti | @gunnarmorling #Infinispan #Debezium Infinispan Client/Server Microservice 1 Microservice 2 hotrod 💖 Rest ... Microservice 3 Data Infinispan Server Cluster Data Data

Slide 25

Slide 25 text

@karesti | @gunnarmorling #Infinispan #Debezium Latency chain Microservice 1 Microservice 2 Microservice 3 Microservice 2 Distributed cache with single owner Data

Slide 26

Slide 26 text

@karesti | @gunnarmorling #Infinispan #Debezium Data Center 1 Data Center 2 Load Balancer APP APP Microservice APP APP Microservice Shared State Shared State Shared State Shared State Customers US Customers EU Data Data

Slide 27

Slide 27 text

@karesti | @gunnarmorling #Infinispan #Debezium Cross Site Replication NYC +Data Data CEN Data LON Data RELAY2 RELAY2 RELAY2

Slide 28

Slide 28 text

https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot

Slide 29

Slide 29 text

@karesti | @gunnarmorling #Infinispan #Debezium A Common Problem: Updating Multiple Resources

Slide 30

Slide 30 text

@karesti | @gunnarmorling #Infinispan #Debezium “Friends don’t let friends do dual-writes!” A Common Problem: Updating Multiple Resources

Slide 31

Slide 31 text

@karesti | @gunnarmorling #Infinispan #Debezium A Better Solution Streaming Changes out of the Database

Slide 32

Slide 32 text

@karesti | @gunnarmorling #Infinispan #Debezium A Better Solution Streaming Changes out of the Database

Slide 33

Slide 33 text

@karesti | @gunnarmorling #Infinispan #Debezium A Better Solution Streaming Changes out of the Database

Slide 34

Slide 34 text

@karesti | @gunnarmorling #Infinispan #Debezium ● CDC for multiple databases ○ Based on transaction logs ○ Snapshotting, Filtering etc. ● Fully open-source, very active community ● Latest version: 1.4 ● Production deployments at multiple companies (e.g. WePay, JW Player, Convoy, Trivago, OYO, BlaBlaCar etc.) Debezium Streaming Changes out of the Database

Slide 35

Slide 35 text

@karesti | @gunnarmorling #Infinispan #Debezium ● Stable ○ MySQL ○ Postgres ○ SQL Server ○ MongoDB ● Incubating ○ Vitess ○ Cassandra ○ DB2 ○ Oracle Debezium Supported Databases

Slide 36

Slide 36 text

@karesti | @gunnarmorling #Infinispan #Debezium Query-based Log-based All data changes are captured - No polling delay or overhead - Transparent to writing applications and models - Can capture deletes and old record state - Simple Installation/Configuration - Debezium Log vs. Query-based CDC

Slide 37

Slide 37 text

@karesti | @gunnarmorling #Infinispan #Debezium Change Data Capture Low-latency Data Streaming Pipelines Source: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake

Slide 38

Slide 38 text

@karesti | @gunnarmorling #Infinispan #Debezium Change Data Capture Liberation for Your Data

Slide 39

Slide 39 text

@karesti | @gunnarmorling #Infinispan #Debezium ■ Propagate data between different services without coupling ■ Each service keeps optimised views locally Microservices Data Exchange

Slide 40

Slide 40 text

@karesti | @gunnarmorling #Infinispan #Debezium http://bit.ly/debezium-outbox-pattern Outbox Pattern

Slide 41

Slide 41 text

@karesti | @gunnarmorling #Infinispan #Debezium Photo: “Strangler vines on trees, seen on the Mount Sorrow hike” by cynren, under CC BY SA 2.0 Mono-to-Micro: Strangler Fig Pattern

Slide 42

Slide 42 text

@karesti | @gunnarmorling #Infinispan #Debezium Customer Mono-to-Micro: Strangler Fig Pattern

Slide 43

Slide 43 text

@karesti | @gunnarmorling #Infinispan #Debezium Mono-to-Micro: Strangler Fig Pattern Customer’ Router CDC Transformation Reads / Writes Reads Customer

Slide 44

Slide 44 text

@karesti | @gunnarmorling #Infinispan #Debezium Customer Router CDC Reads / Writes Reads / Writes CDC Mono-to-Micro: Strangler Fig Pattern

Slide 45

Slide 45 text

@karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams

Slide 46

Slide 46 text

@karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams

Slide 47

Slide 47 text

@karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams

Slide 48

Slide 48 text

@karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams

Slide 49

Slide 49 text

@karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams

Slide 50

Slide 50 text

@karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams

Slide 51

Slide 51 text

@karesti | @gunnarmorling #Infinispan #Debezium http://bit.ly/debezium-auditlogs Auditing with CDC and Kafka Streams

Slide 52

Slide 52 text

https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

@karesti | @gunnarmorling #Infinispan #Debezium ● Fast start-up, low memory consumption ● Developer joy ● Imperative and Reactive ● Best-of-breed libraries ● Run via HotSpot and GraalVM native binaries Quarkus - Supersonic subatomic Java A Stack for building Cloud-native Apps

Slide 55

Slide 55 text

@karesti | @gunnarmorling #Infinispan #Debezium ● Java API for stateful stream processing ● Rich set of operators ● Scaling out to multiple JVMs ● Interactive queries Kafka Streams Streaming Queries on Kafka Topics

Slide 56

Slide 56 text

@karesti | @gunnarmorling #Infinispan #Debezium Data marshalling

Slide 57

Slide 57 text

https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo

Slide 58

Slide 58 text

@karesti | @gunnarmorling #Infinispan #Debezium Infinispan protostream

Slide 59

Slide 59 text

@karesti | @gunnarmorling #Infinispan #Debezium ...Multi-site application with shared system-of-record database ✅ ...With local, denormalized read views (CQRS) ✅ ... Automatically kept in sync after writes ✅ Today’s Mission Explore how to build a... 🤩

Slide 60

Slide 60 text

@karesti | @gunnarmorling #Infinispan #Debezium ● Partial Updates, Multimap support ● Debezium embedded into Infinispan ● Kafka Connect sink connector for Infinispan ○ https://github.com/infinispan/infinispan-kafka ● CDC connector for Infinispan? Outlook

Slide 61

Slide 61 text

@karesti | @gunnarmorling #Infinispan #Debezium ● Infinispan: @infinispan | https://infinispan.org/ ● Debezium: @debezium | https://debezium.io/ ● Demo: https://github.com/debezium/debezium-examples/ → distributed-caching Learn More

Slide 62

Slide 62 text

@karesti | @gunnarmorling #Infinispan #Debezium Q & A Learn More