Change Stream Processing with Apache Flink and Decodable

Change Data Capture (CDC) has earned its popularity in the stream processing world with the rise of Debezium. Apache Flink supports stream processing over change logs natively.

Decodable leverages both open-source projects and offers a service that simplifies CDC processing with rich APIs and SQL. In this webinar, Sharon and Gunnar will demonstrate some use cases around real-time change stream processing with Decodable, including stream joins from multiple sources, detecting critical change events, and more.

Bring all your questions around CDC and stream processing, so we can make this interactive and fun for everyone!

Transcript

1. Change Stream
   Processing with
   Apache Flink
   Sharon Xie, Gunnar Morling
   

   View Slide

2. Today’s Objectives: Learn about…
   01.
   CDC
   What’s change data capture and why
   should I care?
   02.
   CDC + Flink
   What do I gain by integrating change
   data capture with Apache Flink?
   03.
   Demo
   Enough of the talk, show me the
   code!
   

   View Slide

3. @sharon_rxie @gunnarmorling
   Your Presenters
   Sharon Xie Gunnar Morling
   ● Founding engineer of Decodable
   ● 5 years of Flink experience
   ● @sharon_rxie
   ● Open source software engineer
   ● Former lead of Debezium
   ● @gunnarmorling
   

   View Slide

4. 4
   Decodable is a
   real-time data
   stream processing
   platform
   Powered by Apache Flink ®
   

   View Slide

5. @sharon_rxie @gunnarmorling
   Change Data Capture
   What Is It About?
   

   View Slide

6. @sharon_rxie @gunnarmorling
   Change Data Capture
   A Giant Enabler for Your Data
   

   View Slide

7. @sharon_rxie @gunnarmorling
   Change Streams
   Event Format
   ● Encode insert/update/delete operations to a record
   ● Keyed by primary key
   ● Contain both before and after versions
   key: { user_id: 13 } # ← primary key
   value: {
   op: ‘u’, # ← i = insert, u = update, d = delete
   before: { status: ‘active’, ... },
   after: { status: ‘inactive’, ... },
   ...
   }
   

   View Slide

8. Questions
   ● How to integrate data sinks?
   ● How to stream CDC events to consumers?
   ● How to process events?
   

   View Slide

9. @sharon_rxie @gunnarmorling
   Apache Flink
   Stream Processing of Change Data Events
   

   View Slide

10. @sharon_rxie @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events
    

    View Slide

11. @sharon_rxie @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events
    

    View Slide

12. @sharon_rxie @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events
    

    View Slide

13. @sharon_rxie @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events
    https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/asyncio/
    

    View Slide

14. @sharon_rxie @gunnarmorling
    Apache Flink
    Stream Processing of Change Data Events
    

    View Slide

15. Questions
    ● Do I need to program in Java?
    ● Do I need to set up and operate Flink?
    ● How to gain operational insight into my data flows?
    

    View Slide

16. Demo
    Change Stream Processing
    With SQL
    

    View Slide

17. @sharon_rxie @gunnarmorling
    Demo Scenario
    Joins
    

    View Slide

18. @sharon_rxie @gunnarmorling
    Demo Scenario
    Detecting Changes
    

    View Slide

19. @sharon_rxie @gunnarmorling
    Standing on the Shoulders of Giants
    Upstream Open-Source Communities
    https://debezium.io/
    https://flink.apache.org/
    https://ververica.github.io/flink-cdc-connectors/
    Flink
    CDC
    Debezium
    Apache Flink
    Flink CDC Connectors
    

    View Slide

20. @sharon_rxie @gunnarmorling
    CDC allows data to
    be independently
    processed
    Apache Flink
    natively supports
    to process CDC
    events
    Decodable is built on top
    of Flink and offers change
    stream processing as a
    service
    Wrap Up
    

    View Slide

21. decodable.co 2022
    Thanks!
    decodable.co
    @sharon_rxie | @gunnarmorling | @decodableco
    { sharon | gunnar } @ decodable.co
    Free trial
    Twitter
    E-Mail
    

    View Slide

22. @sharon_rxie @gunnarmorling
    Getting Started with Decodable
    • Visit http://decodable.co
    • Start Free http://app.decodable.co
    • Read the docs http://docs.decodable.co
    • Check out the Decodable YouTube Channel
    • Join our community Slack channel
    • Call Saul.alvarez@decodable.co
    

    View Slide

23. 2022
    Powerful Stream Processing,
    In Minutes Not Months.
    decodable.co
    

    View Slide