Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kafka for Rubyists: Monitoring
Search
karol.galanciak
December 20, 2020
Programming
0
170
Kafka for Rubyists: Monitoring
karol.galanciak
December 20, 2020
Tweet
Share
More Decks by karol.galanciak
See All by karol.galanciak
Kafka For Rubyists - Advanced Karafka
azdaroth
0
520
kafka_for_rubyists_advanced_kafka.pdf
azdaroth
0
290
Kafka for Rubyists: Consuming and Performance
azdaroth
0
270
Kafka for Rubyists: Intro to Kafka
azdaroth
0
400
Kafka for Rubyists: Topics and Publishing Messages
azdaroth
0
180
What's New in Rails 5?
azdaroth
0
240
[PL] Od zera do developera
azdaroth
0
370
Introduction to ember-data-partial-model
azdaroth
0
290
The Art Of Self-Defense - How To Protect Yourself From Nils
azdaroth
1
170
Other Decks in Programming
See All in Programming
数十万行のプロジェクトを Scala 2から3に完全移行した
xuwei_k
0
260
menu基盤チームによるGoogle Cloudの活用事例~Application Integration, Cloud Tasks編~
yoshifumi_ishikura
0
110
Webエンジニア主体のモバイルチームの 生産性を高く保つためにやったこと
igreenwood
0
330
今年一番支援させていただいたのは認証系サービスでした
satoshi256kbyte
1
250
103 Early Hints
sugi_0000
1
220
これでLambdaが不要に?!Step FunctionsのJSONata対応について
iwatatomoya
2
3.6k
テストケースの名前はどうつけるべきか?
orgachem
PRO
0
130
フロントエンドのディレクトリ構成どうしてる? Feature-Sliced Design 導入体験談
osakatechlab
8
4.1k
Beyond ORM
77web
1
160
創造的活動から切り拓く新たなキャリア 好きから始めてみる夜勤オペレーターからSREへの転身
yjszk
1
130
Итераторы в Go 1.23: зачем они нужны, как использовать, и насколько они быстрые?
lamodatech
0
700
KMP와 kotlinx.rpc로 서버와 클라이언트 동기화
kwakeuijin
0
140
Featured
See All Featured
For a Future-Friendly Web
brad_frost
175
9.4k
Large-scale JavaScript Application Architecture
addyosmani
510
110k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
Imperfection Machines: The Place of Print at Facebook
scottboms
266
13k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.3k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
17
2.3k
How to train your dragon (web standard)
notwaldorf
88
5.7k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
38
1.9k
Six Lessons from altMBA
skipperchong
27
3.5k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
32
2.7k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
229
52k
Transcript
Kafka & Karafka - Monitoring
Monitoring - Warning - No monitoring? Forget about using Kafka
on production
Monitoring - Kafka cluster itself vs. application-related metrics - Both
are critical - Karafka uses ruby-kafka (versions 1.x) under the hood, which has a great integration with Datadog/Statsd
Kafka Broker metrics - Under-replicated partitions
Kafka Broker metrics - Active Controller Count
Kafka Broker metrics - messages_in_per_sec
Kafka Broker metrics - bytes_in_per_sec
Kafka Broker metrics - bytes_out_per_sec
Kafka Broker metrics - Leader count
Kafka Broker metrics - Offline Partitions Count
Kafka Broker metrics - Replication Max Lag
Kafka Broker metrics - (AWS MSK) data_logs_disk_used
Producer metrics - is stuff working at all? - ruby_kafka.api.calls
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.messages
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.attempts
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.errors
Consumer metrics - does stuff work at all? - ruby_kafka.api.calls
Consumer metrics - does stuff work at all? - ruby_kafka.api.errors
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.lag{*} by
{topic,partition}
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.messages by
{topic,partition}
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.leave_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.sync_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.join_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.fetcher.queue_size
Thanks!