Apache Kafka — The Hard Parts

Slide 1

Slide 1 text

Apache Kafka @duffleit THE HARD PARTS

Slide 2

Slide 2 text

David Leitner @duffleit Coding Architect [email protected] @duﬄeit

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

The Basics Cluster Node A Node B Node C Producer Consumer Consumer Consumer Group Topic Partition A Partition B Partition C Partition A' Partition A'' Partition B' Partition B'' Partition C' Partition C'' Consumer Consumer The number of partitions is the limiting factor for consumer instances. Partition D Partition '' Partition ' User: Bob User: Alice User: Tim selection of partition: = hash(key) % 3 to 4 User: Bob User: Bob User: Bob User: Alice Topic.v2

Slide 9

Slide 9 text

The Basics Cluster Node A Node B Node C Producer Consumer Consumer Consumer Group Topic Partition A Partition B Partition C Partition A' Partition A'' Partition B' Partition B'' Partition C' Partition C'' Consumer Consumer The number of partitions is the limiting factor for consumer instances. Select is wisely & over partition a bit. Partition D Partition '' Partition ' User: Bob User: Alice User: Tim selection of partition: = hash(key) % 3 to 4 User: Bob User: Bob User: Bob User: Alice Topic.v2 Select something that can be devided by multipe numbers. e.g. 6, 12, 24, ...

Slide 10

Slide 10 text

The Basics Topic: UserChanges User: Bob User: Alice User: Tim User: Bob User: Bob log.cleanup.policy: delete

Slide 11

Slide 11 text

The Basics Topic: UserChanges User: Bob User: Alice User: Tim User: Bob User: Bob log.cleanup.policy: delete 2 weeks or some size

Slide 12

Slide 12 text

The Basics Topic: UserChanges User: Bob User: Alice User: Tim User: Bob User: Bob log.cleanup.policy: delete 2 weeks

Slide 13

Slide 13 text

The Basics Topic: UserChanges User: Bob User: Alice User: Tim User: Bob User: Bob log.cleanup.policy: compact we only keep the latest record for a specific key.

Slide 14

Slide 14 text

The Basics Topic: UserChanges User: Bob User: Alice User: Tim User: Bob User: Bob log.cleanup.policy: compact we only keep the latest record for a specific key. How Kafka Stores Data userchanges.active.segment 📄 userchanges.segment.1 userchanges.segment.2 log.segment.bytes = 1GB Active Compaction ⚙ Compaction ⚙

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

The Basics Topic: UserChanges User: Bob User: Alice User: Tim User: Bob User: Bob log.cleanup.policy: compact Tombstone: Tim delete.retention.ms = 1day Slow Consumer needs more than one day to read all events from the topic that starts new. Tombstone: Tim

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

The Basics Cluster Node C Producer Consumer Consumer Consumer Group Topic Partition A Partition B Partition C Partition A' Partition A'' Partition B' Partition B'' Partition C' Partition C'' Node B Node A Node F Node E Node D AZ 1 AZ 2 Partition D Partition E Partition F Partition D' Partition D'' Partition E' Partition E'' Partition F' Partition F'' KIP-36: Rack aware replica assignment

Slide 22

Slide 22 text

Cluster The Basics Node C Producer Consumer Consumer Consumer Group Topic Partition A Partition B Partition C Partition A' Partition A'' Partition B' Partition B'' Partition C' Partition C'' Node B Node A Node F Node E Node D AZ 1 AZ 2 Partition D Partition E Partition F Partition D' Partition D'' Partition E' Partition E'' Partition F' Partition F'' KIP-36: Rack aware replica assignment

Slide 23

Slide 23 text

Multi Region

Slide 24

Slide 24 text

Cluster Multi Region? Node C Producer Consumer Consumer Consumer Group Topic Partition A Partition B Partition C Node B Node A Node F Node E Node D Region 1 Region 2 Partition D' Partition D'' Partition E' Partition E'' Partition F' Partition F'' Partition A' Partition A'' Partition B' Partition B'' Partition C' Partition C'' Partition D Partition E Partition F Usually the latency between multiple regions is to big to span a single cluster over it

Slide 25

Slide 25 text

Cluster Region West Cluster Region East Multi Region? Node C Producer Consumer Consumer Consumer Group Topic Partition A Partition B Partition C Node B Node A Node F Node E Node D Region 1 Region 2 Partition D' Partition D'' Partition E' Partition E'' Partition F' Partition F''

Slide 26

Slide 26 text

Cluster Region West Cluster Region East Multi Region? Node C Producer Consumer Consumer Consumer Group Topic A B C Node B Node A Node F Node E Node D Region 1 Region 2 A B C

Slide 27

Slide 27 text

Cluster Region West Cluster Region East Multi Region? Node C Producer East Consumer Consumer Consumer Group Topic A B C Node B Node A Node F Node E Node D Region 1 Region 2 A B C Producer West

Slide 28

Slide 28 text

Cluster Region West — "west" Cluster Region East — "east" Multi Region? Node C Producer East Consumer Consumer Consumer Group Topic A B C Node B Node A Node F Node E Node D Region 1 Region 2 A B C Producer West Mirror Maker 2 east.C west.C *.C Ordering Guarantees?!

Slide 29

Slide 29 text

@duffleit Order is Guaranteed in a single Partition. Are you sure?

Slide 30

Slide 30 text

@duffleit Producer Partition max.in.flight.requests.per.connection = 5 Message A Message B Message A retry Message B Message A retries = MAX_INT

Slide 31

Slide 31 text

@duffleit Producer Partition max.in.flight.requests.per.connection = 5 Message A Message B Message A retry Message B Message A Legacy Solution: max.in.flight.requests.per.connection = 1 State-of-the-Art Solution: enable.idempotence = true retries = MAX_INT max.in.flight.requests.per.connection = 5 acks = all SEQ#: 1 SEQ#: 2 OutOfOrderSequenceException SEQ#: 2 If you don't want to set your retries to invinite prefer "delivery.timeout.ms" over "retries".

Slide 32

Slide 32 text

Node A Node B Node C Topic Partition A Partition B Partition C Partition C' Partition B'' Partition A' Partition C'' Partition B' Partition A'' Producer acks = none acks = 1 acks = all min.insync.replicas = 3 @duffleit

Slide 33

Slide 33 text

Slide 34

Slide 34 text

min.insync.replicas = 2 Node A Node B Node C Topic Partition A Partition B Partition C Partition C' Partition B'' Partition A' Partition C'' Partition B' Partition A'' Producer acks = all @duffleit CAP Theorem Consistency Availabiltiy Paritioning ❌ min.insync.replicas++ min.insync.replicas--

Slide 35

Slide 35 text

min.insync.replicas = 3 Node A Node B Node C Topic Partition A Partition B Partition C Partition C' Partition B'' Partition A' Partition C'' Partition B' Partition A'' Producer acks = all @duffleit CAP Theorem Consistency Availabiltiy Paritioning ❌ min.insync.replicas++ min.insync.replicas--

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

@duffleit Keep in mind that rack assignment is ingnored for insync replicas. Node C Node B Node A Node D Node E Node F AZ 1 AZ 2 Text . . . . . replicas=6 min.insync.replicas = 4

Slide 40

Slide 40 text

@duffleit Keep in mind that rack assignment is ingnored for insync replicas. Node C Node B Node A Node D Node E Node F AZ 1 AZ 2 Text . . . . replicas=5 min.insync.replicas = 4 fail on > 1

Slide 41

Slide 41 text

@duffleit Keep in mind that rack assignment is ingnored for insync replicas. Node C Node B Node A Node D Node E Node F Node G Node H Node I AZ 1 AZ 2 AZ 3 Text . . . . . . . . replicas=9 min.insync.replicas = 7

Slide 42

Slide 42 text

@duffleit @duffleit Let's talk about Lost Messages

Slide 43

Slide 43 text

@duffleit Partition Message A Consumer Message B Message C enable.auto.commit=true auto.commit.interval.ms=5_SEC

Slide 44

Slide 44 text

@duffleit Partition Message A Consumer Message B Message C enable.auto.commit=true auto.commit.interval.ms=5_SEC Auto-Commit: A,B,C

Slide 45

Slide 45 text

@duffleit Partition Message D Consumer enable.auto.commit=true auto.commit.interval.ms=5_SEC Message A Message B Message C enable.auto.commit=false

Slide 46

Slide 46 text

@duffleit

Slide 47

Slide 47 text

Producer Stream Consistency Kafka, and "Non-Kafka". Transaction to achieve Message

Slide 48

Slide 48 text

Producer Stream Consistency Kafka, and "Non-Kafka". Transaction to achieve Message Message

Slide 49

Slide 49 text

Producer Stream Consistency Kafka, and "Non-Kafka". Transaction to achieve Message Message Producer Consumers Achieve Exactly Once Semantics. Transaction to achieve Message

Slide 50

Slide 50 text

Producer Stream Consistency Kafka, and "Non-Kafka". Transaction to achieve Message Message Producer Consumers Achieve Exactly Once Semantics. Transaction to achieve Message Exactly Once. Atomicity between multipe Topic Operations. Transaction to achieve Transactions Balances Producer Message

Slide 51

Slide 51 text

Slide 52

Slide 52 text

@duffleit Onboarding UserUpdate Stream 💥 ✅

Slide 53

Slide 53 text

@duffleit Onboarding UserUpdate Stream User UserEvents CDC Outbox Pattern ✅

Slide 54

Slide 54 text

@duffleit Onboarding UserUpdate Stream ✅ User Listen to yourself Pattern.

Slide 55

Slide 55 text

@duffleit Onboarding UserUpdated (age: 21) Stream UserUpdated (age: 21) Advertisment User (age: 21) User (age: 21) EventSourcing

Slide 56

Slide 56 text

@duffleit Onboarding UserUpdated (age: 22) Stream UserUpdated (age: 21) Advertisment User (age: 21) User (age: 21) UserUpdated (age: 22) EventSourcing

Slide 57

Slide 57 text

@duffleit Onboarding UserUpdated (age: 22) Stream UserUpdated (age: 21) Advertisment User (age: 22) User (age: 22) UserUpdated (age: 22) UserUpdated (age: 23) EventSourcing

Slide 58

Slide 58 text

@duffleit Onboarding UserUpdated (age: 22) Stream Global EventSourcing UserUpdated (age: 21) Advertisment User (age: 22) User (age: 22) UserUpdated (age: 22) UserUpdated (age: 23) 👻 if often breaks information hiding & data isolation.

Slide 59

Slide 59 text

Stream @duffleit UpdateUser (age: 21) Stream Local EventSourcing UserAgeChanged (age: 21) UserUpdated (age: 21) Onboarding 🔒 👻 💅

Slide 60

Slide 60 text

Producer Producer Stream Consumers Achieve Exactly Once Semantics. Transaction to achieve Consistency Kafka, and "Non-Kafka". Transaction to achieve Atomicity between multipe Topic Operations. Transaction to achieve Transactions Balances Producer Message Message Message Outbox Pattern Listen to Yourself Local Eventsourcing Message

Slide 61

Slide 61 text

@duffleit "Kafka Transactions" Producer Consumers Producer Producer Consumers Consumers Stream Processor Stream Processor Stream Processor enable.idempotence = true isolation.level = read_committed Deduplication Inbox

Slide 62

Slide 62 text

Producer Stream Achieve Exactly Once Semantics. Transaction to achieve Consistency Kafka, and "Non-Kafka". Transaction to achieve Atomicity between multipe Topic Operations. Transaction to achieve Message Outbox Pattern Listen to Yourself Local Eventsourcing Deduplication Inbox Idempotency Producer Consumers Transactions Balances Producer Message Message Message

Slide 63

Slide 63 text

Transfers Payment Service Alice -> Bob Alice -10€ Bob +10€ Transaction_Coordinator

Slide 64

Slide 64 text

Transfers Payment Service Alice -> Bob Alice -10€ Bob +10€ __transaction_state Transaction: ID __consumer_oﬀset payments: 1 P1 P1 P2 Transfers P3 P2 P3 C C C C isolation.level=read_committed

Slide 65

Slide 65 text

Transfers Payment Service Alice -> Bob Alice -10€ __transaction_state Transaction: ID __consumer_oﬀset payments: 1 P1 P1 P2 Transfers P3 P2 isolation.level=read_committed Service A A A Transaction: ID2

Slide 66

Slide 66 text

Producer Stream Achieve Exactly Once Semantics. Transaction to achieve Consistency Kafka, and "Non-Kafka". Transaction to achieve Atomicity between multipe Topic Operations. Transaction to achieve Message Outbox Pattern Listen to Yourself Listen to yourself Deduplication Inbox Idempotency Producer Consumers Transactions Balances Producer Message Message Message Kafka's Exactly-Once Semantics Outbox Pattern

Slide 67

Slide 67 text

@duffleit

Slide 68

Slide 68 text

@duffleit Producer Consumers Topic 📜 📜 Producer Producer Producer Producer Producer Producer Producer Consumers Consumers Consumers Consumers Consumers Consumers Consumers Topic Topic Topic Topic Topic Topic Topic

Slide 69

Slide 69 text

Consumers Consumers Consumers Consumers Consumers Consumers Consumers Producer Producer Producer Producer Producer Producer Producer @duffleit Producer Consumers Topic 📜 📜 Topic Topic Topic Topic Topic Topic Topic

Slide 70

Slide 70 text

@duffleit Producer Consumers Topic Schema Registry Faulty Message Producer Producer Producer Producer Producer Producer Producer Broker Side Validation, FTW

Slide 71

Slide 71 text

@duffleit Producer Consumers Topic Schema Registry Faulty Message Broker Side Validation 🤚 Deserialization on Broker 😱 MagicByte SubjectId Payload ✅ Check if MagicByte Exists. ✅ Check if SubjectId is Valid. ✅ Check if Payload Matches Schema. The more to the right, the more expensive it gets.

Slide 72

Slide 72 text

@duffleit squer.link/broker-side-valdiation-sidecar

Slide 73

Slide 73 text

@duffleit Cluster Node A Node B Node C Go Proxy Go Proxy Go Proxy ⏳ ⏳ ⏳

Slide 74

Slide 74 text

@duffleit Cluster Node A Node B Node C Go Proxy Go Proxy Go Proxy Go Proxy Go Proxy Go Proxy Race Condition We can no longer guarantee ordering.

Slide 75

Slide 75 text

@duffleit squer.link/broker-side-valdiation-sidecar

Slide 76

Slide 76 text

Ok, Lets sum up. @duffleit

Slide 77

Slide 77 text

@duffleit Multi AZ, Multi Region, Multi Cloud Consistency vs. Availability Disable Autocommit! Diﬀerent Options to Achieve Transactional Guarantees in Kafka Broker Side Schema Validation Segment Size Portion Size: "over-partition a bit" and 200+ more Conﬁguration Properties. What we have seen 👀