Ins and Outs of the Outbox Pattern

Slide 1

Slide 1 text

Gunnar Morling @gunnarmorling.dev Ins and Outs of the Outbox Pattern oatsy40 https://flic.kr/p/FvD3mP (CC BY 2.0) 🦋

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

#OutboxPattern · @gunnarmorling.dev Agenda

Slide 4

Slide 4 text

#OutboxPattern · @gunnarmorling.dev ● Technologist at Confluent ● Former project lead of Debezium ● kcctl 🧸, JfrUnit, ModiTect, MapStruct ● One Billion Row Challenge 1⃣🐝🏎 ● Java Champion Gunnar Morling

Slide 5

Slide 5 text

#OutboxPattern · @gunnarmorling.dev The Problem How to… Update a service’s database… …AND notify other services? Consistently.

Slide 6

Slide 6 text

#OutboxPattern · @gunnarmorling.dev Dual Writes… …are prone to inconsistencies

Slide 7

Slide 7 text

The Outbox Pattern Elisabeth Hölzl https://flic.kr/p/m7rKFr (CC BY-SA 2.0)

Slide 8

Slide 8 text

#OutboxPattern · @gunnarmorling.dev Outbox Pattern

Slide 9

Slide 9 text

#OutboxPattern · @gunnarmorling.dev Outbox Pattern

Slide 10

Slide 10 text

#OutboxPattern · @gunnarmorling.dev Outbox Pattern

Slide 11

Slide 11 text

#OutboxPattern · @gunnarmorling.dev Outbox Pattern

Slide 12

Slide 12 text

#OutboxPattern · @gunnarmorling.dev Outbox Pattern The Outbox Table

Slide 13

Slide 13 text

Polling Michael Coghlan https://flic.kr/p/PDwXXS (CC BY-SA 2.0)

Slide 14

Slide 14 text

#OutboxPattern · @gunnarmorling.dev Polling Happy Path

Slide 15

Slide 15 text

#OutboxPattern · @gunnarmorling.dev Polling Ordering Problems

Slide 16

Slide 16 text

#OutboxPattern · @gunnarmorling.dev Polling Missing Events

Slide 17

Slide 17 text

Log-based CDC Barry Silver https://flic.kr/p/8A6XQV (CC BY 2.0)

Slide 18

Slide 18 text

#OutboxPattern · @gunnarmorling.dev Debezium

Slide 19

Slide 19 text

#OutboxPattern · @gunnarmorling.dev Debezium New and Lesser Known Features ● Incremental snapshots ● JDBC sink connector ● Official support for MariaDB ● Re-Select post processor ● WASM-based SMT

Slide 20

Slide 20 text

#OutboxPattern · @gunnarmorling.dev Moving to the Commonhaus Foundation

Slide 21

Slide 21 text

#OutboxPattern · @gunnarmorling.dev Debezium Outbox Event Router

Slide 22

Slide 22 text

#OutboxPattern · @gunnarmorling.dev Debezium Outbox Event Router

Slide 23

Slide 23 text

#OutboxPattern · @gunnarmorling.dev Housekeeping Is Not Needed Actually!

Slide 24

Slide 24 text

#OutboxPattern · @gunnarmorling.dev Postgres-only pg_logical_emit_message()

Slide 25

Slide 25 text

Backﬁlls Jeff Golden https://flic.kr/p/sRGBDX (CC BY-SA 2.0)

Slide 26

Slide 26 text

#OutboxPattern · @gunnarmorling.dev Backﬁlls

Slide 27

Slide 27 text

#OutboxPattern · @gunnarmorling.dev Backﬁlls

Slide 28

Slide 28 text

#OutboxPattern · @gunnarmorling.dev Backﬁlls

Slide 29

Slide 29 text

#OutboxPattern · @gunnarmorling.dev DBLog: A Watermark Based Change-Data-Capture Framework

Slide 30

Slide 30 text

Idempotency formulanone https://flic.kr/p/woErp4 (CC BY-SA 2.0)

Slide 31

Slide 31 text

#OutboxPattern · @gunnarmorling.dev Consumer Idempotency How to Identify Duplicates? ● A unique value ● Not a database sequence ● An increasing value, e.g. PG LSN

Slide 32

Slide 32 text

Discussion Lars K. Jensen https://flic.kr/p/7n4tJL (CC BY-SA 2.0)

Slide 33

Slide 33 text

#OutboxPattern · @gunnarmorling.dev Discussion Database Overhead Doesn’t writing outbox events create database overhead?

Slide 34

Slide 34 text

#OutboxPattern · @gunnarmorling.dev Discussion Database Overhead Doesn’t writing outbox events create database overhead? Yes, but…

Slide 35

Slide 35 text

#OutboxPattern · @gunnarmorling.dev Discussion Latency Doesn’t writing to the database increase end-to-end latency?

Slide 36

Slide 36 text

#OutboxPattern · @gunnarmorling.dev Discussion Latency Doesn’t writing to the database increase end-to-end latency? Yes, but…

Slide 37

Slide 37 text

Alternatives Alexander Svensson https://flic.kr/p/21UKFVe (CC BY 2.0)

Slide 38

Slide 38 text

#OutboxPattern · @gunnarmorling.dev Read Yourself Writing to Kafka ﬁrst ● Con: no synchronous “read-your-own-writes”

Slide 39

Slide 39 text

#OutboxPattern · @gunnarmorling.dev 2PC ● Con: Reduced availability

Slide 40

Slide 40 text

#OutboxPattern · @gunnarmorling.dev Stream Processing ● Con: complexities, e.g. around transactional consistency

Slide 41

Slide 41 text

Summary Jennifer Kramer https://flic.kr/p/24Uqs7 (CC BY 2.0)

Slide 42

Slide 42 text

#OutboxPattern · @gunnarmorling.dev Summary ● Trade-offs every ● Consider pg_logical_emit_message()

Slide 43

Slide 43 text

#OutboxPattern · @gunnarmorling.dev Summary ● Trade-offs every ● Consider pg_logical_emit_message()

Slide 44

Slide 44 text

#OutboxPattern · @gunnarmorling.dev gmorling@conﬂuent.io @gunnarmorling @gunnarmorling.dev morling.dev 📧 Get In Touch

Slide 45

Slide 45 text

No content