Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kafka for Rubyists: Monitoring
Search
karol.galanciak
December 20, 2020
Programming
0
160
Kafka for Rubyists: Monitoring
karol.galanciak
December 20, 2020
Tweet
Share
More Decks by karol.galanciak
See All by karol.galanciak
Kafka For Rubyists - Advanced Karafka
azdaroth
0
500
kafka_for_rubyists_advanced_kafka.pdf
azdaroth
0
280
Kafka for Rubyists: Consuming and Performance
azdaroth
0
260
Kafka for Rubyists: Intro to Kafka
azdaroth
0
390
Kafka for Rubyists: Topics and Publishing Messages
azdaroth
0
170
What's New in Rails 5?
azdaroth
0
220
[PL] Od zera do developera
azdaroth
0
350
Introduction to ember-data-partial-model
azdaroth
0
280
The Art Of Self-Defense - How To Protect Yourself From Nils
azdaroth
1
160
Other Decks in Programming
See All in Programming
複数プロダクトの技術改善・クラウド移行に向き合うチームのフレキシブルなペア・モブプログラミングの実践 / Flexible Pair Programming And Mob Programming
honyanya
0
150
ECS向けのドリフト検知機構を実装してみた
tkikuc
0
270
GitHub Copilot Workspace で我々のアプリ開発がどう変わるのか?
shuyakinjo
0
850
Progressive Web Apps for Rails developers
siaw23
2
530
CSC509 Lecture 02
javiergs
PRO
0
160
AWS CDKを用いたセキュアなCI/CDパイプラインの構築 / Build a secure CI/CD pipeline using AWS CDK
seike460
PRO
3
500
WEBアプリケーションにおけるAWS Lambdaを用いた大規模な非同期処理の実践
delhi09
PRO
7
3.8k
Kotlin Multiplatform at Stable and Beyond (Kotlin Vienna, October 2024)
zsmb
2
320
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
inesmontani
PRO
3
1.1k
[KR] Server Driven Compose With Firebase
skydoves
1
120
NANIMACHI
naokiito
0
930
フロントエンドの現在地とこれから
koba04
6
3.4k
Featured
See All Featured
How to name files
jennybc
77
98k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
109
6.9k
We Have a Design System, Now What?
morganepeng
49
7.1k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
355
29k
No one is an island. Learnings from fostering a developers community.
thoeni
19
2.9k
Practical Orchestrator
shlominoach
185
10k
Facilitating Awesome Meetings
lara
49
6k
StorybookのUI Testing Handbookを読んだ
zakiyama
26
5.1k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
45
4.9k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Code Review Best Practice
trishagee
62
16k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
228
52k
Transcript
Kafka & Karafka - Monitoring
Monitoring - Warning - No monitoring? Forget about using Kafka
on production
Monitoring - Kafka cluster itself vs. application-related metrics - Both
are critical - Karafka uses ruby-kafka (versions 1.x) under the hood, which has a great integration with Datadog/Statsd
Kafka Broker metrics - Under-replicated partitions
Kafka Broker metrics - Active Controller Count
Kafka Broker metrics - messages_in_per_sec
Kafka Broker metrics - bytes_in_per_sec
Kafka Broker metrics - bytes_out_per_sec
Kafka Broker metrics - Leader count
Kafka Broker metrics - Offline Partitions Count
Kafka Broker metrics - Replication Max Lag
Kafka Broker metrics - (AWS MSK) data_logs_disk_used
Producer metrics - is stuff working at all? - ruby_kafka.api.calls
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.messages
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.attempts
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.errors
Consumer metrics - does stuff work at all? - ruby_kafka.api.calls
Consumer metrics - does stuff work at all? - ruby_kafka.api.errors
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.lag{*} by
{topic,partition}
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.messages by
{topic,partition}
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.leave_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.sync_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.join_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.fetcher.queue_size
Thanks!