Change Stream Processing with Apache Flink and Decodable

Change Data Capture (CDC) has earned its popularity in the stream processing world with the rise of Debezium. Apache Flink supports stream processing over change logs natively.

Decodable leverages both open-source projects and offers a service that simplifies CDC processing with rich APIs and SQL. In this webinar, Sharon and Gunnar will demonstrate some use cases around real-time change stream processing with Decodable, including stream joins from multiple sources, detecting critical change events, and more.

Bring all your questions around CDC and stream processing, so we can make this interactive and fun for everyone!

Transcript

1. Change Stream Processing with Apache Flink Sharon Xie, Gunnar Morling
2. Today’s Objectives: Learn about… 01. CDC What’s change data capture and why should I care? 02. CDC + Flink What do I gain by integrating change data capture with Apache Flink? 03. Demo Enough of the talk, show me the code!
3. @sharon_rxie @gunnarmorling Your Presenters Sharon Xie Gunnar Morling • Founding engineer of Decodable • 5 years of Flink experience • @sharon_rxie • Open source software engineer • Former lead of Debezium • @gunnarmorling
4. 4 Decodable is a real-time data stream processing platform Powered by Apache Flink ®
5. @sharon_rxie @gunnarmorling Change Data Capture What Is It About?
6. @sharon_rxie @gunnarmorling Change Data Capture A Giant Enabler for Your Data
7. @sharon_rxie @gunnarmorling Change Streams Event Format • Encode insert/update/delete operations to a record • Keyed by primary key • Contain both before and after versions key: { user_id: 13 } # ← primary key value: { op: ‘u’, # ← i = insert, u = update, d = delete before: { status: ‘active’, ... }, after: { status: ‘inactive’, ... }, ... }
8. Questions • How to integrate data sinks? • How to stream CDC events to consumers? • How to process events?
9. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events
10. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events
11. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events
12. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events
13. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/asyncio/
14. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events
15. Questions • Do I need to program in Java? • Do I need to set up and operate Flink? • How to gain operational insight into my data flows?
16. Demo Change Stream Processing With SQL
17. @sharon_rxie @gunnarmorling Demo Scenario Joins
18. @sharon_rxie @gunnarmorling Demo Scenario Detecting Changes
19. @sharon_rxie @gunnarmorling Standing on the Shoulders of Giants Upstream Open-Source Communities https://debezium.io/ https://flink.apache.org/ https://ververica.github.io/flink-cdc-connectors/ Flink CDC Debezium Apache Flink Flink CDC Connectors
20. @sharon_rxie @gunnarmorling CDC allows data to be independently processed Apache Flink natively supports to process CDC events Decodable is built on top of Flink and offers change stream processing as a service Wrap Up
21. decodable.co 2022 Thanks! decodable.co @sharon_rxie | @gunnarmorling | @decodableco { sharon | gunnar } @ decodable.co Free trial Twitter E-Mail
22. @sharon_rxie @gunnarmorling Getting Started with Decodable • Visit http://decodable.co • Start Free http://app.decodable.co • Read the docs http://docs.decodable.co • Check out the Decodable YouTube Channel • Join our community Slack channel • Call Saul.alvarez@decodable.co
23. 2022 Powerful Stream Processing, In Minutes Not Months. decodable.co