Tweet
Tweet

Slide 1

Slide 1 text

Image © massmatt https://flic.kr/p/25eF9D3 (CC BY 2.0) From Postgres To OpenSearch In No Time Gunnar Morling Software Engineer, Decodable @gunnarmorling

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

From Postgres to OpenSearch | @gunnarmorling Today’s Mission Learn About…

Slide 4

Slide 4 text

From Postgres to OpenSearch | @gunnarmorling ● Software engineer at Decodable ● Former project lead of Debezium ● kcctl 🧸, JfrUnit, ModiTect, MapStruct ● Spec Lead for Bean Validation 2.0 ● Java Champion Gunnar Morling

Slide 5

Slide 5 text

From Postgres to OpenSearch | @gunnarmorling Updating the Search Index One Idea? 🤔

Slide 6

Slide 6 text

© Kai Schreiber https://flic.kr/p/uecg (CC BY-SA 2.0)

Slide 7

Slide 7 text

From Postgres to OpenSearch | @gunnarmorling Debezium Log-Based Change Data Capture

Slide 8

Slide 8 text

From Postgres to OpenSearch | @gunnarmorling Debezium in a Nutshell Open-Source Change Data Capture ● A CDC Platform ○ Based on transaction logs ○ Snapshotting, filtering, etc. ○ Outbox support ○ Web-based UI ● Fully open-source, very active community ● Large production deployments

Slide 9

Slide 9 text

From Postgres to OpenSearch | @gunnarmorling Change Data Capture Liberation for Your Data

Slide 10

Slide 10 text

From Postgres to OpenSearch | @gunnarmorling Change Data Capture Liberation for Your Data

Slide 11

Slide 11 text

From Postgres to OpenSearch | @gunnarmorling ● Core ○ MySQL, MariaDB ○ Postgres ○ SQL Server ○ MongoDB ○ Db2, Informix ○ Oracle ● Community-led: ○ Vitess, Cassandra, Spanner ● External: ScyllaDB, Yugabyte Debezium Supported Databases

Slide 12

Slide 12 text

From Postgres to OpenSearch | @gunnarmorling Debezium: Data Change Events ● Old and new row state ● Metadata on table, TX id, etc. ● Operation type, timestamp

Slide 13

Slide 13 text

From Postgres to OpenSearch | @gunnarmorling Debezium: Data Change Events ● Old and new row state ● Metadata on table, TX id, etc. ● Operation type, timestamp

Slide 14

Slide 14 text

From Postgres to OpenSearch | @gunnarmorling Debezium: Data Change Events ● Old and new row state ● Metadata on table, TX id, etc. ● Operation type, timestamp

Slide 15

Slide 15 text

From Postgres to OpenSearch | @gunnarmorling Becoming the De-Facto CDC Standard https://debezium.io/blog/2021/09/22/deep-dive-into-a-debezium-community-connector-scylla-cdc-source-connector/ Debezium

Slide 16

Slide 16 text

Apache Flink Colin Howley https://flic.kr/p/698F5j (CC BY-ND 2.0)

Slide 17

Slide 17 text

From Postgres to OpenSearch | @gunnarmorling Apache Flink Stateful Computations over Data Streams https://flink.apache.org/

Slide 18

Slide 18 text

From Postgres to OpenSearch | @gunnarmorling ● Real-time reporting/dashboards ● Low-latency alerting, notifications ● Materialized view maintenance, caches ● Real-time cross-database sync, lookup joins, windowed joins, aggregations ● Machine learning: model serving, feature engineering ● Change data capture, data integration Apache Flink Common Use Cases https://flink.apache.org/poweredby.html

Slide 19

Slide 19 text

From Postgres to OpenSearch | @gunnarmorling Apache Flink APIs for Application Development Image source: “Change Data Capture with Flink SQL and Debezium” by Marta Paes at DataEngBytes (https://noti.st/morsapaes/liQzgs/change-data-capture-with-flink-sql-and-debezium)

Slide 20

Slide 20 text

From Postgres to OpenSearch | @gunnarmorling Apache Flink APIs for Application Development Image source: “Change Data Capture with Flink SQL and Debezium” by Marta Paes at DataEngBytes (https://noti.st/morsapaes/liQzgs/change-data-capture-with-flink-sql-and-debezium)

Slide 21

Slide 21 text

From Postgres to OpenSearch | @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 22

Slide 22 text

From Postgres to OpenSearch | @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 23

Slide 23 text

From Postgres to OpenSearch | @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 24

Slide 24 text

From Postgres to OpenSearch | @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 25

Slide 25 text

From Postgres to OpenSearch | @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 26

Slide 26 text

From Postgres to OpenSearch | @gunnarmorling Apache Flink Stream Processing of Change Data Events

Slide 27

Slide 27 text

From Postgres to OpenSearch | @gunnarmorling Debezium and Apache Flink Integration Options

Slide 28

Slide 28 text

Demo © Luke Jones https://flic.kr/p/sEq4MA (CC BY-SA 2.0)

Slide 29

Slide 29 text

From Postgres to OpenSearch | @gunnarmorling Driving Full-Text Search Propagating Joined Data to OpenSearch

Slide 30

Slide 30 text

Demo © Luke Jones https://flic.kr/p/sEq4MA (CC BY-SA 2.0)

Slide 31

Slide 31 text

From Postgres to OpenSearch | @gunnarmorling Nested Data Structures UDFs to the Rescue

Slide 32

Slide 32 text

From Postgres to OpenSearch | @gunnarmorling Nested Data Structures UDFs to the Rescue

Slide 33

Slide 33 text

Demo © Luke Jones https://flic.kr/p/sEq4MA (CC BY-SA 2.0)

Slide 34

Slide 34 text

From Postgres to OpenSearch | @gunnarmorling Nested Data Structures UDFs to the Rescue

Slide 35

Slide 35 text

From Postgres to OpenSearch | @gunnarmorling Nested Data Structures UDFs to the Rescue

Slide 36

Slide 36 text

From Postgres to OpenSearch | @gunnarmorling Nested Data Structures UDFs to the Rescue https://www.youtube.com/@decodable

Slide 37

Slide 37 text

From Postgres to OpenSearch | @gunnarmorling Ingest Once... ...Process Multiple Times

Slide 38

Slide 38 text

From Postgres to OpenSearch | @gunnarmorling Transactional Aggregation Correlating Events From Same Transaction

Slide 39

Slide 39 text

From Postgres to OpenSearch | @gunnarmorling Wrap-Up

Slide 40

Slide 40 text

From Postgres to OpenSearch | @gunnarmorling ● Debezium: Real-time change event streams for your data ● Debezium and Apache Flink: Power house of change stream processing ○ Data Integration ○ Data Cleansing ○ Denormalization ○ Aggregations ○ Pattern Matching Take Aways 🤩

Slide 41

Slide 41 text

From Postgres to OpenSearch | @gunnarmorling ● Provisioning and updating infrastructure ● Deployment and (auto-)scaling ● Observability ● State management ● Schema management and inference ● Developer experience ● CI/CD ● Security and access control Towards Production What To Consider

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

From Postgres to OpenSearch | @gunnarmorling ● Debezium: @debezium | https://debezium.io/ ● Apache Flink: @ApacheFlink | https://flink.apache.org/ ● Getting started with Flink: github.com/decodableco/examples → flink-learn Learn More

Slide 44

Slide 44 text

From Postgres to OpenSearch | @gunnarmorling gunnar@decodable.co @gunnarmorling 📧 Thank You! Q & A

Slide 45

Slide 45 text

No content