Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Kafka for Rubyists: Monitoring
Search
Sponsored
·
SiteGround - Reliable hosting with speed, security, and support you can count on.
→
karol.galanciak
December 20, 2020
Programming
290
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Kafka for Rubyists: Monitoring
karol.galanciak
December 20, 2020
Transcript
25 slides
Kafka & Karafka - Monitoring
Monitoring - Warning - No monitoring? Forget about using Kafka
on production
Monitoring - Kafka cluster itself vs. application-related metrics - Both
are critical - Karafka uses ruby-kafka (versions 1.x) under the hood, which has a great integration with Datadog/Statsd
Kafka Broker metrics - Under-replicated partitions
Kafka Broker metrics - Active Controller Count
Kafka Broker metrics - messages_in_per_sec
Kafka Broker metrics - bytes_in_per_sec
Kafka Broker metrics - bytes_out_per_sec
Kafka Broker metrics - Leader count
Kafka Broker metrics - Offline Partitions Count
Kafka Broker metrics - Replication Max Lag
Kafka Broker metrics - (AWS MSK) data_logs_disk_used
Producer metrics - is stuff working at all? - ruby_kafka.api.calls
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.messages
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.attempts
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.errors
Consumer metrics - does stuff work at all? - ruby_kafka.api.calls
Consumer metrics - does stuff work at all? - ruby_kafka.api.errors
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.lag{*} by
{topic,partition}
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.messages by
{topic,partition}
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.leave_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.sync_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.join_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.fetcher.queue_size
Thanks!
More Decks by karol.galanciak
See All by karol.galanciak
CPTO - The Role of the Future in Startups an Scaleups (and How to Become One)
azdaroth
0
23
Kafka For Rubyists - Advanced Karafka
azdaroth
0
640
kafka_for_rubyists_advanced_kafka.pdf
azdaroth
0
400
Kafka for Rubyists: Consuming and Performance
azdaroth
0
390
Kafka for Rubyists: Intro to Kafka
azdaroth
0
520
Kafka for Rubyists: Topics and Publishing Messages
azdaroth
0
290
What's New in Rails 5?
azdaroth
0
370
[PL] Od zera do developera
azdaroth
0
490
Introduction to ember-data-partial-model
azdaroth
0
440
Other Decks in Programming
See All in Programming
React Native経験者が Flutterを30⽇間触って驚いたこと
3saito
0
190
Ruby から Emacs を操作したい
phigasui
0
120
flutter_test のその先へ — Flutter アプリの LLM 機能に「評価」という品質保証を
kosukesaigusa
0
200
Python Lambdaにおけるテストしやすい設計とAI支援の活用 / Designing Testable Python Lambda Functions with AI
kitano_yuichi
1
330
仕様駆動開発入門 (何を作るかを先に決めて渡す) | 目指せ Kiro マスター 05
inariku
PRO
2
170
Enforcing Your Architecture with Unit Tests in the AI Era (Swift Connection X)
steliosf
PRO
1
180
Secure Your Coding Agent Like It’s Malware (v1.5) 🇬🇧 @Code.talks 2026
richargh
0
160
SODA - FACT BOOK(EN)
sodainc
1
250
React Native에 현대적인 빌드 도구 더하기: Rollipop 구현기
leegeunhyeok
0
180
Seven New Principles of Software Engineering
yegor256
0
500
AWS CDKを理解するために公式の定義を深掘って解説してみる / Understanding AWS CDK by Digging into Its Official Definition
akihisaikeda
3
310
複数ドメインを一つのアプリに統合するFlutterの設計と運用_FlutterKaigi2026登壇資料(須永高弘)
takahirosunaga
0
190
Featured
See All Featured
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
17k
SEO for Brand Visibility & Recognition
aleyda
0
4.9k
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
akihiro_kokubo
PRO
74
43k
Un-Boring Meetings
codingconduct
0
500
Discover your Explorer Soul
emna__ayadi
2
1.3k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
marketingsoph
0
470
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
9
1.7k
WCS-LA-2024
lcolladotor
0
1k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
12
1.4k
How to train your dragon (web standard)
notwaldorf
96
6.9k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
47
8.4k
Abbi's Birthday
coloredviolet
5
11k