Slide 1

Slide 1 text

Change Stream Processing with Apache Flink Sharon Xie, Gunnar Morling

Slide 2

Slide 2 text

Today’s Objectives: Learn about… 01. CDC What’s change data capture and why should I care? 02. CDC + Flink What do I gain by integrating change data capture with Apache Flink? 03. Demo Enough of the talk, show me the code!

Slide 3

Slide 3 text

@sharon_rxie @gunnarmorling Your Presenters Sharon Xie Gunnar Morling ● Founding engineer of Decodable ● 5 years of Flink experience ● @sharon_rxie ● Open source software engineer ● Former lead of Debezium ● @gunnarmorling

Slide 4

Slide 4 text

4 Decodable is a real-time data stream processing platform Powered by Apache Flink ®

Slide 5

Slide 5 text

@sharon_rxie @gunnarmorling Change Data Capture What Is It About?

Slide 6

Slide 6 text

@sharon_rxie @gunnarmorling Change Data Capture A Giant Enabler for Your Data

Slide 7

Slide 7 text

@sharon_rxie @gunnarmorling Change Streams Event Format ● Encode insert/update/delete operations to a record ● Keyed by primary key ● Contain both before and after versions key: { user_id: 13 } # ← primary key value: { op: ‘u’, # ← i = insert, u = update, d = delete before: { status: ‘active’, ... }, after: { status: ‘inactive’, ... }, ... }

Slide 8

Slide 8 text

Questions ● How to integrate data sinks? ● How to stream CDC events to consumers? ● How to process events?

Slide 9

Slide 9 text

@sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 10

Slide 10 text

@sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 11

Slide 11 text

@sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 12

Slide 12 text

@sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 13

Slide 13 text

@sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/asyncio/

Slide 14

Slide 14 text

@sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 15

Slide 15 text

Questions ● Do I need to program in Java? ● Do I need to set up and operate Flink? ● How to gain operational insight into my data flows?

Slide 16

Slide 16 text

Demo Change Stream Processing With SQL

Slide 17

Slide 17 text

@sharon_rxie @gunnarmorling Demo Scenario Joins

Slide 18

Slide 18 text

@sharon_rxie @gunnarmorling Demo Scenario Detecting Changes

Slide 19

Slide 19 text

@sharon_rxie @gunnarmorling Standing on the Shoulders of Giants Upstream Open-Source Communities https://debezium.io/ https://flink.apache.org/ https://ververica.github.io/flink-cdc-connectors/ Flink CDC Debezium Apache Flink Flink CDC Connectors

Slide 20

Slide 20 text

@sharon_rxie @gunnarmorling CDC allows data to be independently processed Apache Flink natively supports to process CDC events Decodable is built on top of Flink and offers change stream processing as a service Wrap Up

Slide 21

Slide 21 text

decodable.co 2022 Thanks! decodable.co @sharon_rxie | @gunnarmorling | @decodableco { sharon | gunnar } @ decodable.co Free trial Twitter E-Mail

Slide 22

Slide 22 text

@sharon_rxie @gunnarmorling Getting Started with Decodable • Visit http://decodable.co • Start Free http://app.decodable.co • Read the docs http://docs.decodable.co • Check out the Decodable YouTube Channel • Join our community Slack channel • Call [email protected]

Slide 23

Slide 23 text

2022 Powerful Stream Processing, In Minutes Not Months. decodable.co