Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kafka for Rubyists: Monitoring
Search
karol.galanciak
December 20, 2020
Programming
0
160
Kafka for Rubyists: Monitoring
karol.galanciak
December 20, 2020
Tweet
Share
More Decks by karol.galanciak
See All by karol.galanciak
Kafka For Rubyists - Advanced Karafka
azdaroth
0
490
kafka_for_rubyists_advanced_kafka.pdf
azdaroth
0
270
Kafka for Rubyists: Consuming and Performance
azdaroth
0
250
Kafka for Rubyists: Intro to Kafka
azdaroth
0
380
Kafka for Rubyists: Topics and Publishing Messages
azdaroth
0
170
What's New in Rails 5?
azdaroth
0
220
[PL] Od zera do developera
azdaroth
0
330
Introduction to ember-data-partial-model
azdaroth
0
260
The Art Of Self-Defense - How To Protect Yourself From Nils
azdaroth
1
160
Other Decks in Programming
See All in Programming
リハビリmruby
kishima
1
160
GraphQL はいいぞ! ~Laravel で学ぶ GraphQL 入門~
azuki
1
160
Rust.Nagoya #1
codemountains
0
170
DMMプラットフォームにおけるTiDBの導入から運用まで
pospome
7
3k
Webエディタライブラリ 「CodeMirror」から学ぶ Webアプリ開発のテクニック
ryosukeigarashi
0
250
小さな開発会社を作った理由
polidog
0
1.9k
Introduction to GitOps
hwchiu
0
110
Rustのweb開発を助ける 便利なツール紹介
yuki0418
1
190
[After Kotlin Fest 2024 LT Night @ Sansan] もっともっとKotlinを好きになる!K2 Compiler Pluginで遊んでみよう!
kitakkun
2
260
CSC307 Lecture 08
javiergs
PRO
0
330
実用的かつリーズナブルな 「Azure × Gemini × LINE」~キャラクターBot 実装ライブデモ~
tomodo_ysys
1
170
初心者がおさえておきたいAWS CDKのベストプラクティス 2024
konokenj
15
7.3k
Featured
See All Featured
A Philosophy of Restraint
colly
200
16k
The Cult of Friendly URLs
andyhume
75
5.9k
Principles of Awesome APIs and How to Build Them.
keavy
124
16k
Bash Introduction
62gerente
607
210k
KATA
mclloyd
20
13k
Practical Orchestrator
shlominoach
185
10k
RailsConf 2023
tenderlove
16
720
Infographics Made Easy
chrislema
238
18k
How to name files
jennybc
67
96k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
10 Git Anti Patterns You Should be Aware of
lemiorhan
652
58k
Fireside Chat
paigeccino
25
2.8k
Transcript
Kafka & Karafka - Monitoring
Monitoring - Warning - No monitoring? Forget about using Kafka
on production
Monitoring - Kafka cluster itself vs. application-related metrics - Both
are critical - Karafka uses ruby-kafka (versions 1.x) under the hood, which has a great integration with Datadog/Statsd
Kafka Broker metrics - Under-replicated partitions
Kafka Broker metrics - Active Controller Count
Kafka Broker metrics - messages_in_per_sec
Kafka Broker metrics - bytes_in_per_sec
Kafka Broker metrics - bytes_out_per_sec
Kafka Broker metrics - Leader count
Kafka Broker metrics - Offline Partitions Count
Kafka Broker metrics - Replication Max Lag
Kafka Broker metrics - (AWS MSK) data_logs_disk_used
Producer metrics - is stuff working at all? - ruby_kafka.api.calls
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.messages
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.attempts
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.errors
Consumer metrics - does stuff work at all? - ruby_kafka.api.calls
Consumer metrics - does stuff work at all? - ruby_kafka.api.errors
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.lag{*} by
{topic,partition}
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.messages by
{topic,partition}
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.leave_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.sync_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.join_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.fetcher.queue_size
Thanks!