Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kafka for Rubyists: Monitoring
Search
karol.galanciak
December 20, 2020
Programming
0
200
Kafka for Rubyists: Monitoring
karol.galanciak
December 20, 2020
Tweet
Share
More Decks by karol.galanciak
See All by karol.galanciak
Kafka For Rubyists - Advanced Karafka
azdaroth
0
550
kafka_for_rubyists_advanced_kafka.pdf
azdaroth
0
310
Kafka for Rubyists: Consuming and Performance
azdaroth
0
290
Kafka for Rubyists: Intro to Kafka
azdaroth
0
420
Kafka for Rubyists: Topics and Publishing Messages
azdaroth
0
200
What's New in Rails 5?
azdaroth
0
270
[PL] Od zera do developera
azdaroth
0
400
Introduction to ember-data-partial-model
azdaroth
0
320
The Art Of Self-Defense - How To Protect Yourself From Nils
azdaroth
1
190
Other Decks in Programming
See All in Programming
The Efficiency Paradox and How to Save Yourself and the World
hollycummins
0
110
国漢文混用体からHolloまで
minhee
1
180
タイムゾーンの奥地は思ったよりも闇深いかもしれない
suguruooki
1
640
複雑なフォームの jotai 設計 / Designing jotai(state) for Complex Forms #layerx_frontend
izumin5210
3
390
Optimizing JRuby 10
headius
0
300
Code smarter, not harder - How AI Coding Tools Boost Your Productivity | Webinar 2025
danielsogl
0
130
メモリウォールを超えて:キャッシュメモリ技術の進歩
kawayu
0
1.9k
SEAL - Dive into the sea of search engines - Symfony Live Berlin 2025
alexanderschranz
1
130
Amazon CloudWatchの地味だけど強力な機能紹介!
itotsum
0
150
Compose Hot Reload is here, stop re-launching your apps! (Android Makers 2025)
zsmb
1
500
リストビュー画面UX改善の振り返り
splcywolf
0
140
サービスクラスのありがたみを発見したときの思い出 #phpcon_odawara
77web
4
640
Featured
See All Featured
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
21k
Statistics for Hackers
jakevdp
798
220k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
13
670
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
23
2.6k
Raft: Consensus for Rubyists
vanstee
137
6.9k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
Why You Should Never Use an ORM
jnunemaker
PRO
55
9.3k
Designing for Performance
lara
608
69k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
227
22k
For a Future-Friendly Web
brad_frost
176
9.7k
Transcript
Kafka & Karafka - Monitoring
Monitoring - Warning - No monitoring? Forget about using Kafka
on production
Monitoring - Kafka cluster itself vs. application-related metrics - Both
are critical - Karafka uses ruby-kafka (versions 1.x) under the hood, which has a great integration with Datadog/Statsd
Kafka Broker metrics - Under-replicated partitions
Kafka Broker metrics - Active Controller Count
Kafka Broker metrics - messages_in_per_sec
Kafka Broker metrics - bytes_in_per_sec
Kafka Broker metrics - bytes_out_per_sec
Kafka Broker metrics - Leader count
Kafka Broker metrics - Offline Partitions Count
Kafka Broker metrics - Replication Max Lag
Kafka Broker metrics - (AWS MSK) data_logs_disk_used
Producer metrics - is stuff working at all? - ruby_kafka.api.calls
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.messages
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.attempts
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.errors
Consumer metrics - does stuff work at all? - ruby_kafka.api.calls
Consumer metrics - does stuff work at all? - ruby_kafka.api.errors
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.lag{*} by
{topic,partition}
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.messages by
{topic,partition}
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.leave_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.sync_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.join_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.fetcher.queue_size
Thanks!