Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Change Stream Processing with Apache Flink and Decodable

Change Stream Processing with Apache Flink and Decodable

Change Data Capture (CDC) has earned its popularity in the stream processing world with the rise of Debezium. Apache Flink supports stream processing over change logs natively.

Decodable leverages both open-source projects and offers a service that simplifies CDC processing with rich APIs and SQL. In this webinar, Sharon and Gunnar will demonstrate some use cases around real-time change stream processing with Decodable, including stream joins from multiple sources, detecting critical change events, and more.

Bring all your questions around CDC and stream processing, so we can make this interactive and fun for everyone!

Gunnar Morling

November 17, 2022
Tweet

More Decks by Gunnar Morling

Other Decks in Programming

Transcript

  1. Change Stream Processing with Apache Flink Sharon Xie, Gunnar Morling

  2. Today’s Objectives: Learn about… 01. CDC What’s change data capture

    and why should I care? 02. CDC + Flink What do I gain by integrating change data capture with Apache Flink? 03. Demo Enough of the talk, show me the code!
  3. @sharon_rxie @gunnarmorling Your Presenters Sharon Xie Gunnar Morling • Founding

    engineer of Decodable • 5 years of Flink experience • @sharon_rxie • Open source software engineer • Former lead of Debezium • @gunnarmorling
  4. 4 Decodable is a real-time data stream processing platform Powered

    by Apache Flink ®
  5. @sharon_rxie @gunnarmorling Change Data Capture What Is It About?

  6. @sharon_rxie @gunnarmorling Change Data Capture A Giant Enabler for Your

    Data
  7. @sharon_rxie @gunnarmorling Change Streams Event Format • Encode insert/update/delete operations

    to a record • Keyed by primary key • Contain both before and after versions key: { user_id: 13 } # ← primary key value: { op: ‘u’, # ← i = insert, u = update, d = delete before: { status: ‘active’, ... }, after: { status: ‘inactive’, ... }, ... }
  8. Questions • How to integrate data sinks? • How to

    stream CDC events to consumers? • How to process events?
  9. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

  10. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

  11. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

  12. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

  13. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

    https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/asyncio/
  14. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

  15. Questions • Do I need to program in Java? •

    Do I need to set up and operate Flink? • How to gain operational insight into my data flows?
  16. Demo Change Stream Processing With SQL

  17. @sharon_rxie @gunnarmorling Demo Scenario Joins

  18. @sharon_rxie @gunnarmorling Demo Scenario Detecting Changes

  19. @sharon_rxie @gunnarmorling Standing on the Shoulders of Giants Upstream Open-Source

    Communities https://debezium.io/ https://flink.apache.org/ https://ververica.github.io/flink-cdc-connectors/ Flink CDC Debezium Apache Flink Flink CDC Connectors
  20. @sharon_rxie @gunnarmorling CDC allows data to be independently processed Apache

    Flink natively supports to process CDC events Decodable is built on top of Flink and offers change stream processing as a service Wrap Up
  21. decodable.co 2022 Thanks! decodable.co @sharon_rxie | @gunnarmorling | @decodableco {

    sharon | gunnar } @ decodable.co Free trial Twitter E-Mail
  22. @sharon_rxie @gunnarmorling Getting Started with Decodable • Visit http://decodable.co •

    Start Free http://app.decodable.co • Read the docs http://docs.decodable.co • Check out the Decodable YouTube Channel • Join our community Slack channel • Call [email protected]
  23. 2022 Powerful Stream Processing, In Minutes Not Months. decodable.co