Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How Plumbr uses Kafka
Search
Nikita Salnikov-Tarnovski
February 04, 2018
Programming
0
100
How Plumbr uses Kafka
Nikita Salnikov-Tarnovski
February 04, 2018
Tweet
Share
More Decks by Nikita Salnikov-Tarnovski
See All by Nikita Salnikov-Tarnovski
Project clarity - random rant from an old engineer
nikem
0
93
Introduction to Druid
nikem
0
860
Deceived by monitoring
nikem
0
69
10% Happier
nikem
0
72
Where is my memory
nikem
0
460
Heap, off you go
nikem
0
1.2k
First steps in GC tuning
nikem
0
1.6k
I bet you have a memory leak
nikem
1
170
Plumbing Memory Leaks
nikem
1
140
Other Decks in Programming
See All in Programming
Range on Rails ―「多重範囲型」という新たな選択肢が、複雑ロジックを劇的にシンプルにしたワケ
rizap_tech
0
6.7k
pnpm に provenance のダウングレード を検出する PR を出してみた
ryo_manba
1
150
CSC305 Lecture 09
javiergs
PRO
0
300
AI駆動で0→1をやって見えた光と伸びしろ
passion0102
1
780
20251016_Rails News ~Rails 8.1の足音を聴く~
morimorihoge
2
670
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
580
Cursorハンズオン実践!
eltociear
2
1.2k
TFLintカスタムプラグインで始める Terraformコード品質管理
bells17
2
350
組込みだけじゃない!TinyGo で始める無料クラウド開発入門
otakakot
2
360
技術的負債の正体を知って向き合う
irof
0
230
実践Claude Code:20の失敗から学ぶAIペアプログラミング
takedatakashi
18
7.6k
What's new in Spring Modulith?
olivergierke
1
160
Featured
See All Featured
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
9
930
The World Runs on Bad Software
bkeepers
PRO
72
11k
Product Roadmaps are Hard
iamctodd
PRO
55
11k
Rails Girls Zürich Keynote
gr2m
95
14k
Designing for humans not robots
tammielis
254
26k
KATA
mclloyd
PRO
32
15k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3.1k
Gamification - CAS2011
davidbonilla
81
5.5k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.6k
A better future with KSS
kneath
239
18k
Principles of Awesome APIs and How to Build Them.
keavy
127
17k
How to Ace a Technical Interview
jacobian
280
24k
Transcript
Eating Kafka Nikita Salnikov-Tarnovski @iNikem
Intro to Kafka
What is Kafka • Distributed streaming platform • It lets
you publish and subscribe to streams of records • It lets you store streams of records in a fault-tolerant way.
What is Kafka • Kafka runs as a cluster on
one or more servers. • The Kafka cluster stores streams of records in categories called topics. • Each record consists of a key, a value, and a timestamp.
Four APIs http://kafka.apache.org/documentation/
Append log http://kafka.apache.org/documentation/
Brokers • Several brokers form a cluster • Coordinated with
Zookeeper • All partitions are distributed among brokers
Producers • Producer sends record to a topic • Based
on a key, partition is chosen • Leader broker is found • Wait for requested acks
Fast writes • Brokers cheat and don’t write to disk
• They write to disk cache • And let OS care about flushing to disk
Replication • Each topic can be replicated among brokers •
So for each partition there are X copies • Brokers just consume messages from leader
Consumer groups (c) Confluent
Consumer rebalance (c) Confluent
Commit • Consumer has to commit offsets he consumed •
You have to decide, when and how!
Delivery semantics • At least once • At most once
• Exactly once
Kafka Connect • Off-the-shelf solution to pipe data to or
from Kafka • E.g. DB, Elasticsearch, files, etc…
Kafka Streams • DSL and platform for writing data processing
streams • If you squint enough, very similar to Java8 streams and Fork-Join pool • But across multiple jvms and servers
Kafka in Plumbr
Kafka cluster • 5 brokers • 2x replication • 20T
data for last 90 days • Inflow ~125G per day
Data processing pipeline
Spring Cloud Stream • Greatly simplifies development of Kafka based
apps • Couple of annotations and data flows :)
Solving performance problems is hard. We don’t think it needs
to be. @JavaPlumbr/@iNikem http://plumbr.eu