Kafka, the hard parts - Speaker Deck

Slide 1

Slide 1 text

Kafka The Hard Parts Chris Keathley / @ChrisKeathley / keathley.io

Slide 2

Slide 2 text

Kafka is great

Slide 3

Slide 3 text

Kafka is just a log

Slide 4

Slide 4 text

https://ﬂic.kr/p/9aXr88

Slide 5

Slide 5 text

https://ﬂic.kr/p/9aXr88 Kafka

Slide 6

Slide 6 text

Kafka https://ﬂic.kr/p/9aXr88 (metaphor)

Slide 7

Slide 7 text

Log aggregation Analytics and activity tracking Queuing ETL Messaging Stream Processing Kafka Uses

Slide 8

Slide 8 text

Event Sourcing

Slide 9

Slide 9 text

Log aggregation Analytics and activity tracking Queuing ETL Messaging Stream Processing Kafka Uses

Slide 10

Slide 10 text

https://ﬂic.kr/p/hrrbVx

Slide 11

Slide 11 text

https://ﬂic.kr/p/hrrbVx (still a metaphor) Kafka

Slide 12

Slide 12 text

Large consequences for failure

Slide 13

Slide 13 text

Joke about mr. glass

Slide 14

Slide 14 text

Joke about mr. glass

Slide 15

Slide 15 text

Iteration Is Hard

Slide 16

Slide 16 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 17

Slide 17 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 18

Slide 18 text

Topic

Slide 19

Slide 19 text

Topic

Slide 20

Slide 20 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5

Slide 21

Slide 21 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5

Slide 22

Slide 22 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5

Slide 23

Slide 23 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Written to the File system

Slide 24

Slide 24 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5

Slide 25

Slide 25 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Messages are ordered

Slide 26

Slide 26 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5

Slide 27

Slide 27 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5

Slide 28

Slide 28 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Consumer

Slide 29

Slide 29 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Consumer

Slide 30

Slide 30 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Consumer

Slide 31

Slide 31 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Consumer

Slide 32

Slide 32 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Consumer

Slide 33

Slide 33 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Consumer Consumer

Slide 34

Slide 34 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Consumer Consumer

Slide 35

Slide 35 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Consumer Consumer

Slide 36

Slide 36 text

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Consumer Consumer

Slide 37

Slide 37 text

Topic

Slide 38

Slide 38 text

Topic Topic Topic Topic Broker

Slide 39

Slide 39 text

Broker Broker Broker

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Replication Leader

Slide 42

Slide 42 text

Clients Java Client librdkafka

Slide 43

Slide 43 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 44

Slide 44 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 45

Slide 45 text

Order is important User Events

Slide 46

Slide 46 text

Order is important User Events

Slide 47

Slide 47 text

Order is important Follow

Slide 48

Slide 48 text

Order is important Follow

Slide 49

Slide 49 text

Order is important Follow Message

Slide 50

Slide 50 text

Order is important Follow Message Unfollow

Slide 51

Slide 51 text

Order is important Follow Message Unfollow Causal

Slide 52

Slide 52 text

Order is important Follow Message Unfollow Consumer

Slide 53

Slide 53 text

Order is important Follow Message Unfollow Consumer

Slide 54

Slide 54 text

Order is important Follow Message Unfollow Consumer

Slide 55

Slide 55 text

Order is important Follow Message Unfollow

Slide 56

Slide 56 text

Order is important Follow Message Unfollow Consumer

Slide 57

Slide 57 text

Order is important Follow Message Unfollow Consumer

Slide 58

Slide 58 text

Order is important Follow Message Unfollow Consumer

Slide 59

Slide 59 text

Group records based on order

Slide 60

Slide 60 text

Partitioner to_int(hash(key)) % partitions

Slide 61

Slide 61 text

Partitioner to_int(hash(user_id)) % partitions

Slide 62

Slide 62 text

Follow Message Unfollow Grouping Consumers

Slide 63

Slide 63 text

Follow Message Unfollow Causal Grouping Consumers

Slide 64

Slide 64 text

Follow Message Unfollow Grouping Consumers Follow Processor Message Processor

Slide 65

Slide 65 text

Follow Message Unfollow Grouping Consumers Follow Processor Message Processor

Slide 66

Slide 66 text

Follow Message Unfollow Grouping Consumers Follow Processor Message Processor

Slide 67

Slide 67 text

Follow Message Unfollow Grouping Consumers User event processor

Slide 68

Slide 68 text

Follow Message Unfollow Grouping Consumers User event processor

Slide 69

Slide 69 text

Follow Message Unfollow Grouping Consumers User event processor

Slide 70

Slide 70 text

User Events Create pipelines User event processor Messages

Slide 71

Slide 71 text

User Events Create pipelines User event processor Messages Consumes

Slide 72

Slide 72 text

User Events Create pipelines User event processor Messages Consumes Produces

Slide 73

Slide 73 text

"Commander: Better Distributed Applications through CQRS and Event Sourcing" by Bobby Calderwood https://youtu.be/B1-gS0oEtYc

Slide 74

Slide 74 text

The less dependence you can have between consumers the better

Slide 75

Slide 75 text

Random partitioning is best if you can avoid ordering

Slide 76

Slide 76 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 77

Slide 77 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 78

Slide 78 text

Errors have the potential to wreck your day

Slide 79

Slide 79 text

Consumer Errors

Slide 80

Slide 80 text

Consumer Errors

Slide 81

Slide 81 text

Consumer Errors

Slide 82

Slide 82 text

Consumer Errors

Slide 83

Slide 83 text

Consumer Errors Blocking the head of the line

Slide 84

Slide 84 text

Consumer What should we do? Errors

Slide 85

Slide 85 text

Non-Blocking vs. Blocking

Slide 86

Slide 86 text

Non-Blocking vs. Blocking

Slide 87

Slide 87 text

Non-Blocking Errors Consumer 42 1337 “Robert’);drop table students;—”

Slide 88

Slide 88 text

Non-Blocking Errors Consumer 42 1337 “Robert’);drop table students;—” What do we do?

Slide 89

Slide 89 text

Non-Blocking Errors Consumer

Slide 90

Slide 90 text

Non-Blocking Errors Consumer

Slide 91

Slide 91 text

Non-Blocking Errors Consumer Error Topic

Slide 92

Slide 92 text

Non-Blocking Errors Consumer

Slide 93

Slide 93 text

Non-Blocking Errors Consumer

Slide 94

Slide 94 text

Non-Blocking Errors Consumer

Slide 95

Slide 95 text

Non-Blocking vs. Blocking

Slide 96

Slide 96 text

Non-Blocking vs. Blocking

Slide 97

Slide 97 text

Blocking Errors Database Consumer

Slide 98

Slide 98 text

Blocking Errors Database Consumer Process messages Store Information

Slide 99

Slide 99 text

Blocking Errors Database Consumer

Slide 100

Slide 100 text

Blocking Errors Database Consumer

Slide 101

Slide 101 text

Blocking Errors Database Consumer What do we do?

Slide 102

Slide 102 text

Blocking Errors Database Consumer Retry

Slide 103

Slide 103 text

Blocking Errors Database Consumer Send alerts

Slide 104

Slide 104 text

Skip non-blocking errors & Retry blocking errors

Slide 105

Slide 105 text

Design errors out of existence

Slide 106

Slide 106 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 107

Slide 107 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 108

Slide 108 text

Delivery Guarantees

Slide 109

Slide 109 text

Computer A Communication is hard Computer B What time is it?

Slide 110

Slide 110 text

Computer A Communication is hard Computer B

Slide 111

Slide 111 text

Computer A Communication is hard Computer B Did you get it?

Slide 112

Slide 112 text

Computer A Communication is hard Computer B How about now?

Slide 113

Slide 113 text

Computer A Communication is hard Computer B Now?

Slide 114

Slide 114 text

0 <= 1 <= n Delivery At least once At most once Impossible-ish

Slide 115

Slide 115 text

Consumers should *ALWAYS* assume “At Least Once”

Slide 116

Slide 116 text

The Joys of Functional Programming

Slide 117

Slide 117 text

No content

Slide 118

Slide 118 text

You

Slide 119

Slide 119 text

You Functional Programming

Slide 120

Slide 120 text

Immutability and Idempotence

Slide 121

Slide 121 text

Immutability: An immutable object is an object whose state cannot be modiﬁed after it is created.

Slide 122

Slide 122 text

Idempotence: …the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application.

Slide 123

Slide 123 text

Idempotence: Execute the same operation more than once but only see the eﬀect once.

Slide 124

Slide 124 text

Idempotent Operations

Slide 125

Slide 125 text

Counting comments comment comment comment increment 1

Slide 126

Slide 126 text

Counting comments comment comment comment increment 1

Slide 127

Slide 127 text

Counting comments comment comment comment increment 2

Slide 128

Slide 128 text

Counting comments comment comment comment increment 2

Slide 129

Slide 129 text

Counting comments comment comment comment increment 3

Slide 130

Slide 130 text

Counting comments comment comment comment increment 3 Some Error

Slide 131

Slide 131 text

Counting comments comment comment comment increment 3

Slide 132

Slide 132 text

Counting comments comment comment comment increment 3

Slide 133

Slide 133 text

Counting comments comment comment comment increment 4

Slide 134

Slide 134 text

Counting comments comment comment comment increment 4

Slide 135

Slide 135 text

Counting comments comment comment comment increment 5

Slide 136

Slide 136 text

Counting comments comment comment comment increment 5

Slide 137

Slide 137 text

Counting comments comment comment comment increment 6

Slide 138

Slide 138 text

Kafka Record { data: {}, type: “comment.created”, }

Slide 139

Slide 139 text

Kafka Record { data: {}, type: “comment.created”, msg_id: UUIDv4 }

Slide 140

Slide 140 text

Kafka Record { data: {}, type: “comment.created”, msg_id: UUIDv4 } Used for managing idempotence

Slide 141

Slide 141 text

Counting comments comment comment comment increment 1

Slide 142

Slide 142 text

Counting comments comment comment comment Set.add(id) id: 1 id: 2 id: 3 (1)

Slide 143

Slide 143 text

Counting comments comment comment comment Set.add(id) id: 1 id: 2 id: 3 (1)

Slide 144

Slide 144 text

Counting comments comment comment comment Set.add(id) id: 1 id: 2 id: 3 (1, 2)

Slide 145

Slide 145 text

Counting comments comment comment comment Set.add(id) id: 1 id: 2 id: 3 (1, 2)

Slide 146

Slide 146 text

Counting comments comment comment comment Set.add(id) id: 1 id: 2 id: 3 (1, 2, 3)

Slide 147

Slide 147 text

Counting comments comment comment comment Set.add(id) id: 1 id: 2 id: 3 (1, 2, 3) Some Error

Slide 148

Slide 148 text

Counting comments comment comment comment id: 1 id: 2 id: 3 Set.add(id) (1, 2, 3)

Slide 149

Slide 149 text

Counting comments comment comment comment id: 1 id: 2 id: 3 Set.add(id) (1, 2, 3)

Slide 150

Slide 150 text

Counting comments comment comment comment id: 1 id: 2 id: 3 Set.add(id) (1, 2, 3)

Slide 151

Slide 151 text

Counting comments (1, 2, 3)

Slide 152

Slide 152 text

Counting comments cardinality(1, 2, 3)

Slide 153

Slide 153 text

Counting comments cardinality(1, 2, 3) => 3

Slide 154

Slide 154 text

Idempotent Side-Eﬀects

Slide 155

Slide 155 text

smtp send_email Sending Emails email id: 1 email id: 2 email id: 3

Slide 156

Slide 156 text

smtp send_email Sending Emails email id: 1 email id: 2 email id: 3 What do we do if this fails?

Slide 157

Slide 157 text

smtp send_email Sending Emails email id: 1 email id: 2 email id: 3 Send at most once

Slide 158

Slide 158 text

smtp send_email Sending Emails email id: 1

Slide 159

Slide 159 text

Cache send_email Sending Emails email id: 1 smtp

Slide 160

Slide 160 text

Cache send_email Sending Emails email id: 1 smtp id?(1)

Slide 161

Slide 161 text

Cache send_email Sending Emails email id: 1 smtp id?(1) If id exists then skip it

Slide 162

Slide 162 text

Cache send_email Sending Emails email id: 1 smtp

Slide 163

Slide 163 text

Cache send_email Sending Emails email id: 1 smtp add(1)

Slide 164

Slide 164 text

Cache send_email Sending Emails email id: 1 smtp

Slide 165

Slide 165 text

Cache send_email Sending Emails email id: 1 smtp

Slide 166

Slide 166 text

Cache send_email Sending Emails email id: 1 smtp

Slide 167

Slide 167 text

send_email Sending Emails email id: 1

Slide 168

Slide 168 text

send_email Sending Emails email id: 1 If we see this message again move it to an audit topic

Slide 169

Slide 169 text

send_email Sending Emails If we see this message again move it to an audit topic email id: 1

Slide 170

Slide 170 text

send_email Sending Emails

Slide 171

Slide 171 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 172

Slide 172 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 173

Slide 173 text

User Events Teams User event processor Messages Notifications Notiﬁcations Notification Sender

Slide 174

Slide 174 text

User Events Teams User event processor Messages Notifications Notiﬁcations Notification Sender Teams

Slide 175

Slide 175 text

Data is the language of the system

Slide 176

Slide 176 text

{ msg_id: "8700635f-1802-417e-89e7-595ad3600104", type: "comment.created", data: { user_id: 1234, msg: "This is a super fun conference!" } } Data payloads

Slide 177

Slide 177 text

{ msg_id: String, type: String, data: { user_id: Integer, msg: String } } Data payloads

Slide 178

Slide 178 text

{ msg_id: String, type: String, data: { user_id: Integer, msg: String } } Data payloads None of this tells you anything useful about your data

Slide 179

Slide 179 text

{ msg_id: String, type: String, data: { user_id: Integer, msg: String } } Data payloads What do we do when these things change?

Slide 180

Slide 180 text

{ msg_id: String, type: String, data: { user_id: String, msg: String } } Data payloads What do we do when these things change?

Slide 181

Slide 181 text

{ msg_id: String, type: String, data: { user_id: String, msg: String } } Data payloads Lets just use versions!

Slide 182

Slide 182 text

{ msg_id: String, type: String, data: { user_id: String, msg: String } } Data payloads Lets just use versions! (spoiler: this isn’t great)

Slide 183

Slide 183 text

{ msg_id: String, type: String, data: { user_id: String, msg: String } } Data payloads

Slide 184

Slide 184 text

{ msg_id: String, type: String, data: { user_id: String, msg: String }, meta: { version: 2 } } Data payloads

Slide 185

Slide 185 text

Data Versions Consumer v1 v1 v1 v1 v2

Slide 186

Slide 186 text

Data Versions Consumer v1 v1 v1 v1 v2 This consumer needs to understand both versions

Slide 187

Slide 187 text

Data Versions Consumer v1 v1 v1 v1 v2 This team needs to know to make these changes

Slide 188

Slide 188 text

Versioning is broken

Slide 189

Slide 189 text

(sem)Versioning is broken

Slide 190

Slide 190 text

Change Growth Breakage

Slide 191

Slide 191 text

Change Growth Breakage Never do this

Slide 192

Slide 192 text

Growing schemas should be the default

Slide 193

Slide 193 text

{ msg_id: String, type: String, data: { user_id: String, msg: String } } Data payloads

Slide 194

Slide 194 text

{ msg_id: String, type: String, data: { user_id: Integer, msg: String } } Data payloads What are these?

Slide 195

Slide 195 text

Dependent Types

Slide 196

Slide 196 text

{ msg_id: String, type: String, data: { user_id: Integer, msg: String } } Data payloads What are these?

Slide 197

Slide 197 text

Norm

Slide 198

Slide 198 text

{ msg_id: String, type: String, data: { user_id: String, msg: String } } Data payloads

Slide 199

Slide 199 text

UUID = string? & re_matches?(/^[0-9A-F]{8}-[0-9A-F] {4}-4[0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i) ) CommentCreated = schema{ req :msg_id, UUID req :type, lit(“comment.created”) req :data, schema { req :user_id, integer? | UUID req :msg, string? } } Data payloads

Slide 200

Slide 200 text

json = {type: “comment.created”, msg: “Hello world”} Norm.decode(CommentEvent, json) => {:ok, data} Norm.decode(CommentEvent, {}) => {:error, errors}  Norm.explain(CommentEvent, {}) => "In :msg_id, val: {} fails spec: required In :type, val: {} fails spec: required In :data, val: {} fails spec: required" Data payloads

Slide 201

Slide 201 text

Norm is built for extensibility

Slide 202

Slide 202 text

CommentEvent = schema{ req :type, lit(“comment.created”) req :msg, string? } json = { type: “comment.created”, msg: “Hello world”, data: { msg: “Hello world” } } Norm.decode(CommentEvent, json) => {:ok, data} Norm is extensible

Slide 203

Slide 203 text

Slide 204

Slide 204 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 205

Slide 205 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 206

Slide 206 text

Property Based Testing

Slide 207

Slide 207 text

Property based testing Database Consumer

Slide 208

Slide 208 text

Property based testing Database Consumer id: 1 id: 2 id: 3 id: 1

Slide 209

Slide 209 text

Property based testing Database Consumer id: 1 id: 2 id: 3 id: 1 Information should end up here

Slide 210

Slide 210 text

Property based testing Database Consumer id: 1 id: 2 id: 3 id: 1 Some combination of these messages causes a failure

Slide 211

Slide 211 text

Property based testing Database id: 1 id: 1 Consumer

Slide 212

Slide 212 text

Property based testing Database id: 1 id: 1 Looks like we aren’t handling duplicates correctly Consumer

Slide 213

Slide 213 text

Property based testing Database id: 1 id: 1 Consumer

Slide 214

Slide 214 text

Property based testing Database Consumer id: 1 id: 1 Deterministically fail this connection

Slide 215

Slide 215 text

Chaos Engineering

Slide 216

Slide 216 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 217

Slide 217 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes

Slide 218

Slide 218 text

Monitoring vs. Observability

Slide 219

Slide 219 text

Monitoring: Figuring out that there’s a problem

Slide 220

Slide 220 text

Observability: Determining what the problem is.

Slide 221

Slide 221 text

Goal: Detect lagging or blocked consumers

Slide 222

Slide 222 text

Wisen

Slide 223

Slide 223 text

Wisen User Events User Consumer

Slide 224

Slide 224 text

metadata topic Wisen User Events Checkpoints its position in the log to an offset topic User Consumer

Slide 225

Slide 225 text

Wisen metadata topic Wisen User Consumer User Events

Slide 226

Slide 226 text

Wisen metadata topic Wisen User Consumer User Events Compares farthest offset from checkpoints over a time-window

Slide 227

Slide 227 text

Wisen user_consumer_errors Wisen User Consumer User Events

Slide 228

Slide 228 text

Wisen user_consumer_errors Wisen User Consumer User Events

Slide 229

Slide 229 text

Wisen user_consumer_errors Wisen User Consumer User Events Alert if we see a rise in errors

Slide 230

Slide 230 text

Other useful metrics: Median and Tail latencies Internal buﬀers DB/Cache/RPC latencies

Slide 231

Slide 231 text

OpenTracing

Slide 232

Slide 232 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Monitoring Capacity Planning #hottakes

Slide 233

Slide 233 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Monitoring Capacity Planning #hottakes

Slide 234

Slide 234 text

This has to be done up-front

Slide 235

Slide 235 text

Calculating partions messages in the system = arrival rate * mean time in system

Slide 236

Slide 236 text

Calculating partions Desired throughput / measured throughput on one partition => partitions needed

Slide 237

Slide 237 text

Calculating partions partitions < 100 x brokers x replication factor source: https://www.conﬂuent.io/blog/how-choose-number-topics-partitions-kafka-cluster

Slide 238

Slide 238 text

Increasing partitions is tricky if you rely on ordering

Slide 239

Slide 239 text

to_int(hash(user_id)) % partitions

Slide 240

Slide 240 text

to_int(hash(user_id)) % partitions Existing data is not reshuffled if partitions are increased

Slide 241

Slide 241 text

Data is not forever.

Slide 242

Slide 242 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Monitoring Capacity Planning #hottakes

Slide 243

Slide 243 text

Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems and the joys of functional programming Data Validation Monitoring Capacity Planning #hottakes