Speaker Deck
Sign in
Sign up
for free
Kafka, the hard parts
Chris Keathley
January 10, 2019
Programming
1
350
Kafka, the hard parts
This talk tries to summarize a lot of the lessons I've learned building systems on kafka.
Chris Keathley
January 10, 2019
Tweet
Share
Want more?
keathley
5
180
keathley
2
290
keathley
5
670
reverentgeek
4
230
rmw
1
110
skipperchong
0
97
carmenhchung
5
200
productmarketing
2
240
clairelew
3
470
thoeni
1
120
jcasabona
4
210
cherdarchuk
60
250k
Transcript
1
.
Kafka The Hard Parts Chris Keathley / @ChrisKeathley / keathley.io
2
.
Kafka is great
3
.
Kafka is just a log
4
.
https://flic.kr/p/9aXr88
5
.
https://flic.kr/p/9aXr88 Kafka
6
.
Kafka https://flic.kr/p/9aXr88 (metaphor)
7
.
Log aggregation Analytics and activity tracking Queuing ETL Messaging Stream
Processing Kafka Uses
8
.
Event Sourcing
9
.
Log aggregation Analytics and activity tracking Queuing ETL Messaging Stream
Processing Kafka Uses
10
.
https://flic.kr/p/hrrbVx
11
.
https://flic.kr/p/hrrbVx (still a metaphor) Kafka
12
.
Large consequences for failure
13
.
Joke about mr. glass
14
.
Joke about mr. glass
15
.
Iteration Is Hard
16
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
17
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
18
.
Topic
19
.
Topic
20
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
21
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
22
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
23
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Written to the File system
24
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
25
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Messages are ordered
26
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
27
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
28
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Consumer
29
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Consumer
30
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Consumer
31
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Consumer
32
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Consumer
33
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Consumer Consumer
34
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Consumer Consumer
35
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Consumer Consumer
36
.
Partition 1 Partition 2 Partition 3 Partition 4 Partition 5
Consumer Consumer
37
.
Topic
38
.
Topic Topic Topic Topic Broker
39
.
Broker Broker Broker
40
.
None
41
.
Replication Leader
42
.
Clients Java Client librdkafka
43
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
44
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
45
.
Order is important User Events
46
.
Order is important User Events
47
.
Order is important Follow
48
.
Order is important Follow
49
.
Order is important Follow Message
50
.
Order is important Follow Message Unfollow
51
.
Order is important Follow Message Unfollow Causal
52
.
Order is important Follow Message Unfollow Consumer
53
.
Order is important Follow Message Unfollow Consumer
54
.
Order is important Follow Message Unfollow Consumer
55
.
Order is important Follow Message Unfollow
56
.
Order is important Follow Message Unfollow Consumer
57
.
Order is important Follow Message Unfollow Consumer
58
.
Order is important Follow Message Unfollow Consumer
59
.
Group records based on order
60
.
Partitioner to_int(hash(key)) % partitions
61
.
Partitioner to_int(hash(user_id)) % partitions
62
.
Follow Message Unfollow Grouping Consumers
63
.
Follow Message Unfollow Causal Grouping Consumers
64
.
Follow Message Unfollow Grouping Consumers Follow Processor Message Processor
65
.
Follow Message Unfollow Grouping Consumers Follow Processor Message Processor
66
.
Follow Message Unfollow Grouping Consumers Follow Processor Message Processor
67
.
Follow Message Unfollow Grouping Consumers User event processor
68
.
Follow Message Unfollow Grouping Consumers User event processor
69
.
Follow Message Unfollow Grouping Consumers User event processor
70
.
User Events Create pipelines User event processor Messages
71
.
User Events Create pipelines User event processor Messages Consumes
72
.
User Events Create pipelines User event processor Messages Consumes Produces
73
.
"Commander: Better Distributed Applications through CQRS and Event Sourcing" by
Bobby Calderwood https://youtu.be/B1-gS0oEtYc
74
.
The less dependence you can have between consumers the better
75
.
Random partitioning is best if you can avoid ordering
76
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
77
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
78
.
Errors have the potential to wreck your day
79
.
Consumer Errors
80
.
Consumer Errors
81
.
Consumer Errors
82
.
Consumer Errors
83
.
Consumer Errors Blocking the head of the line
84
.
Consumer What should we do? Errors
85
.
Non-Blocking vs. Blocking
86
.
Non-Blocking vs. Blocking
87
.
Non-Blocking Errors Consumer 42 1337 “Robert’);drop table students;—”
88
.
Non-Blocking Errors Consumer 42 1337 “Robert’);drop table students;—” What do
we do?
89
.
Non-Blocking Errors Consumer
90
.
Non-Blocking Errors Consumer
91
.
Non-Blocking Errors Consumer Error Topic
92
.
Non-Blocking Errors Consumer
93
.
Non-Blocking Errors Consumer
94
.
Non-Blocking Errors Consumer
95
.
Non-Blocking vs. Blocking
96
.
Non-Blocking vs. Blocking
97
.
Blocking Errors Database Consumer
98
.
Blocking Errors Database Consumer Process messages Store Information
99
.
Blocking Errors Database Consumer
100
.
Blocking Errors Database Consumer
101
.
Blocking Errors Database Consumer What do we do?
102
.
Blocking Errors Database Consumer Retry
103
.
Blocking Errors Database Consumer Send alerts
104
.
Skip non-blocking errors & Retry blocking errors
105
.
Design errors out of existence
106
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
107
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
108
.
Delivery Guarantees
109
.
Computer A Communication is hard Computer B What time is
it?
110
.
Computer A Communication is hard Computer B
111
.
Computer A Communication is hard Computer B Did you get
it?
112
.
Computer A Communication is hard Computer B How about now?
113
.
Computer A Communication is hard Computer B Now?
114
.
0 <= 1 <= n Delivery At least once At
most once Impossible-ish
115
.
Consumers should *ALWAYS* assume “At Least Once”
116
.
The Joys of Functional Programming
117
.
None
118
.
You
119
.
You Functional Programming
120
.
Immutability and Idempotence
121
.
Immutability: An immutable object is an object whose state cannot
be modified after it is created.
122
.
Idempotence: …the property of certain operations in mathematics and computer
science whereby they can be applied multiple times without changing the result beyond the initial application.
123
.
Idempotence: Execute the same operation more than once but only
see the effect once.
124
.
Idempotent Operations
125
.
Counting comments comment comment comment increment 1
126
.
Counting comments comment comment comment increment 1
127
.
Counting comments comment comment comment increment 2
128
.
Counting comments comment comment comment increment 2
129
.
Counting comments comment comment comment increment 3
130
.
Counting comments comment comment comment increment 3 Some Error
131
.
Counting comments comment comment comment increment 3
132
.
Counting comments comment comment comment increment 3
133
.
Counting comments comment comment comment increment 4
134
.
Counting comments comment comment comment increment 4
135
.
Counting comments comment comment comment increment 5
136
.
Counting comments comment comment comment increment 5
137
.
Counting comments comment comment comment increment 6
138
.
Kafka Record { data: {}, type: “comment.created”, }
139
.
Kafka Record { data: {}, type: “comment.created”, msg_id: UUIDv4 }
140
.
Kafka Record { data: {}, type: “comment.created”, msg_id: UUIDv4 }
Used for managing idempotence
141
.
Counting comments comment comment comment increment 1
142
.
Counting comments comment comment comment Set.add(id) id: 1 id: 2
id: 3 (1)
143
.
Counting comments comment comment comment Set.add(id) id: 1 id: 2
id: 3 (1)
144
.
Counting comments comment comment comment Set.add(id) id: 1 id: 2
id: 3 (1, 2)
145
.
Counting comments comment comment comment Set.add(id) id: 1 id: 2
id: 3 (1, 2)
146
.
Counting comments comment comment comment Set.add(id) id: 1 id: 2
id: 3 (1, 2, 3)
147
.
Counting comments comment comment comment Set.add(id) id: 1 id: 2
id: 3 (1, 2, 3) Some Error
148
.
Counting comments comment comment comment id: 1 id: 2 id:
3 Set.add(id) (1, 2, 3)
149
.
Counting comments comment comment comment id: 1 id: 2 id:
3 Set.add(id) (1, 2, 3)
150
.
Counting comments comment comment comment id: 1 id: 2 id:
3 Set.add(id) (1, 2, 3)
151
.
Counting comments (1, 2, 3)
152
.
Counting comments cardinality(1, 2, 3)
153
.
Counting comments cardinality(1, 2, 3) => 3
154
.
Idempotent Side-Effects
155
.
smtp send_email Sending Emails email id: 1 email id: 2
email id: 3
156
.
smtp send_email Sending Emails email id: 1 email id: 2
email id: 3 What do we do if this fails?
157
.
smtp send_email Sending Emails email id: 1 email id: 2
email id: 3 Send at most once
158
.
smtp send_email Sending Emails email id: 1
159
.
Cache send_email Sending Emails email id: 1 smtp
160
.
Cache send_email Sending Emails email id: 1 smtp id?(1)
161
.
Cache send_email Sending Emails email id: 1 smtp id?(1) If
id exists then skip it
162
.
Cache send_email Sending Emails email id: 1 smtp
163
.
Cache send_email Sending Emails email id: 1 smtp add(1)
164
.
Cache send_email Sending Emails email id: 1 smtp
165
.
Cache send_email Sending Emails email id: 1 smtp
166
.
Cache send_email Sending Emails email id: 1 smtp
167
.
send_email Sending Emails email id: 1
168
.
send_email Sending Emails email id: 1 If we see this
message again move it to an audit topic
169
.
send_email Sending Emails If we see this message again move
it to an audit topic email id: 1
170
.
send_email Sending Emails
171
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
172
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
173
.
User Events Teams User event processor Messages Notifications Notifications Notification
Sender
174
.
User Events Teams User event processor Messages Notifications Notifications Notification
Sender Teams
175
.
Data is the language of the system
176
.
{ msg_id: "8700635f-1802-417e-89e7-595ad3600104", type: "comment.created", data: { user_id: 1234, msg:
"This is a super fun conference!" } } Data payloads
177
.
{ msg_id: String, type: String, data: { user_id: Integer, msg:
String } } Data payloads
178
.
{ msg_id: String, type: String, data: { user_id: Integer, msg:
String } } Data payloads None of this tells you anything useful about your data
179
.
{ msg_id: String, type: String, data: { user_id: Integer, msg:
String } } Data payloads What do we do when these things change?
180
.
{ msg_id: String, type: String, data: { user_id: String, msg:
String } } Data payloads What do we do when these things change?
181
.
{ msg_id: String, type: String, data: { user_id: String, msg:
String } } Data payloads Lets just use versions!
182
.
{ msg_id: String, type: String, data: { user_id: String, msg:
String } } Data payloads Lets just use versions! (spoiler: this isn’t great)
183
.
{ msg_id: String, type: String, data: { user_id: String, msg:
String } } Data payloads
184
.
{ msg_id: String, type: String, data: { user_id: String, msg:
String }, meta: { version: 2 } } Data payloads
185
.
Data Versions Consumer v1 v1 v1 v1 v2
186
.
Data Versions Consumer v1 v1 v1 v1 v2 This consumer
needs to understand both versions
187
.
Data Versions Consumer v1 v1 v1 v1 v2 This team
needs to know to make these changes
188
.
Versioning is broken
189
.
(sem)Versioning is broken
190
.
Change Growth Breakage
191
.
Change Growth Breakage Never do this
192
.
Growing schemas should be the default
193
.
{ msg_id: String, type: String, data: { user_id: String, msg:
String } } Data payloads
194
.
{ msg_id: String, type: String, data: { user_id: Integer, msg:
String } } Data payloads What are these?
195
.
Dependent Types
196
.
{ msg_id: String, type: String, data: { user_id: Integer, msg:
String } } Data payloads What are these?
197
.
Norm
198
.
{ msg_id: String, type: String, data: { user_id: String, msg:
String } } Data payloads
199
.
UUID = string? & re_matches?(/^[0-9A-F]{8}-[0-9A-F] {4}-4[0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i) ) CommentCreated = schema{
req :msg_id, UUID req :type, lit(“comment.created”) req :data, schema { req :user_id, integer? | UUID req :msg, string? } } Data payloads
200
.
json = {type: “comment.created”, msg: “Hello world”} Norm.decode(CommentEvent, json) =>
{:ok, data} Norm.decode(CommentEvent, {}) => {:error, errors} Norm.explain(CommentEvent, {}) => "In :msg_id, val: {} fails spec: required In :type, val: {} fails spec: required In :data, val: {} fails spec: required" Data payloads
201
.
Norm is built for extensibility
202
.
CommentEvent = schema{ req :type, lit(“comment.created”) req :msg, string? }
json = { type: “comment.created”, msg: “Hello world”, data: { msg: “Hello world” } } Norm.decode(CommentEvent, json) => {:ok, data} Norm is extensible
203
.
CommentEvent = schema{ req :type, lit(“comment.created”) req :msg, string? }
json = { type: “comment.created”, msg: “Hello world”, data: { msg: “Hello world” } } Norm.decode(CommentEvent, json) => {:ok, data} Norm is extensible This will still get passed through
204
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
205
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
206
.
Property Based Testing
207
.
Property based testing Database Consumer
208
.
Property based testing Database Consumer id: 1 id: 2 id:
3 id: 1
209
.
Property based testing Database Consumer id: 1 id: 2 id:
3 id: 1 Information should end up here
210
.
Property based testing Database Consumer id: 1 id: 2 id:
3 id: 1 Some combination of these messages causes a failure
211
.
Property based testing Database id: 1 id: 1 Consumer
212
.
Property based testing Database id: 1 id: 1 Looks like
we aren’t handling duplicates correctly Consumer
213
.
Property based testing Database id: 1 id: 1 Consumer
214
.
Property based testing Database Consumer id: 1 id: 1 Deterministically
fail this connection
215
.
Chaos Engineering
216
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
217
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Finding Errors Monitoring Capacity Planning #hottakes
218
.
Monitoring vs. Observability
219
.
Monitoring: Figuring out that there’s a problem
220
.
Observability: Determining what the problem is.
221
.
Goal: Detect lagging or blocked consumers
222
.
Wisen
223
.
Wisen User Events User Consumer
224
.
metadata topic Wisen User Events Checkpoints its position in the
log to an offset topic User Consumer
225
.
Wisen metadata topic Wisen User Consumer User Events
226
.
Wisen metadata topic Wisen User Consumer User Events Compares farthest
offset from checkpoints over a time-window
227
.
Wisen user_consumer_errors Wisen User Consumer User Events
228
.
Wisen user_consumer_errors Wisen User Consumer User Events
229
.
Wisen user_consumer_errors Wisen User Consumer User Events Alert if we
see a rise in errors
230
.
Other useful metrics: Median and Tail latencies Internal buffers DB/Cache/RPC
latencies
231
.
OpenTracing
232
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Monitoring Capacity Planning #hottakes
233
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Monitoring Capacity Planning #hottakes
234
.
This has to be done up-front
235
.
Calculating partions messages in the system = arrival rate *
mean time in system
236
.
Calculating partions Desired throughput / measured throughput on one partition
=> partitions needed
237
.
Calculating partions partitions < 100 x brokers x replication factor
source: https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster
238
.
Increasing partitions is tricky if you rely on ordering
239
.
to_int(hash(user_id)) % partitions
240
.
to_int(hash(user_id)) % partitions Existing data is not reshuffled if partitions
are increased
241
.
Data is not forever.
242
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Monitoring Capacity Planning #hottakes
243
.
Lets talk about… Kafka Terminology Maintaining Order Errors Distributed Systems
and the joys of functional programming Data Validation Monitoring Capacity Planning #hottakes
244
.
CQRS & Event Sourcing
245
.
Don’t rush to democratize your data
246
.
Embrace data and design
247
.
Go forth and build awesome stuff!
248
.
Thanks Chris Keathley / @ChrisKeathley / keathley.io
Chris Keathley
keathley
reverentgeek
rmw
skipperchong
carmenhchung
productmarketing
clairelew
thoeni
jcasabona
cherdarchuk
![UUID = string? & re_matches?(/^[0-9A-F]{8}-[0-9A-F] {4}-4[0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i) ) CommentCreated = schema{](https://files.speakerdeck.com/presentations/0abb7b76c2444f248d61ff0ac32843f1/slide_198.jpg){kind=link}
