Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kafka for Rubyists: Monitoring
Search
karol.galanciak
December 20, 2020
Programming
0
210
Kafka for Rubyists: Monitoring
karol.galanciak
December 20, 2020
Tweet
Share
More Decks by karol.galanciak
See All by karol.galanciak
Kafka For Rubyists - Advanced Karafka
azdaroth
0
570
kafka_for_rubyists_advanced_kafka.pdf
azdaroth
0
330
Kafka for Rubyists: Consuming and Performance
azdaroth
0
310
Kafka for Rubyists: Intro to Kafka
azdaroth
0
440
Kafka for Rubyists: Topics and Publishing Messages
azdaroth
0
220
What's New in Rails 5?
azdaroth
0
280
[PL] Od zera do developera
azdaroth
0
410
Introduction to ember-data-partial-model
azdaroth
0
340
The Art Of Self-Defense - How To Protect Yourself From Nils
azdaroth
1
200
Other Decks in Programming
See All in Programming
可変性を制する設計: 構造と振る舞いから考える概念モデリングとその実装
a_suenami
2
550
プロダクトという一杯を作る - プロダクトチームが味の責任を持つまでの煮込み奮闘記
hiliteeternal
0
180
テストから始めるAgentic Coding 〜Claude Codeと共に行うTDD〜 / Agentic Coding starts with testing
rkaga
16
6k
NEWT Backend Evolution
xpromx
1
150
QA x AIエコシステム段階構築作戦
osu
0
140
「App Intent」よくわからんけどすごい!
rinngo0302
1
120
ZeroETLで始めるDynamoDBとS3の連携
afooooil
0
120
Googleの新しいコーディングAIエージェントJulesを使ってみた
tonionagauzzi
0
120
構文解析器入門
ydah
7
1.8k
Claude Code派?Gemini CLI派? みんなで比較LT会!_20250716
junholee
1
700
Vibe Codingの幻想を超えて-生成AIを現場で使えるようにするまでの泥臭い話.ai
fumiyakume
18
9.2k
リッチエディターを安全に開発・運用するために
unachang113
1
240
Featured
See All Featured
A Modern Web Designer's Workflow
chriscoyier
695
190k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
How STYLIGHT went responsive
nonsquared
100
5.6k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Raft: Consensus for Rubyists
vanstee
140
7k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.2k
Fireside Chat
paigeccino
37
3.5k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Designing for humans not robots
tammielis
253
25k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Transcript
Kafka & Karafka - Monitoring
Monitoring - Warning - No monitoring? Forget about using Kafka
on production
Monitoring - Kafka cluster itself vs. application-related metrics - Both
are critical - Karafka uses ruby-kafka (versions 1.x) under the hood, which has a great integration with Datadog/Statsd
Kafka Broker metrics - Under-replicated partitions
Kafka Broker metrics - Active Controller Count
Kafka Broker metrics - messages_in_per_sec
Kafka Broker metrics - bytes_in_per_sec
Kafka Broker metrics - bytes_out_per_sec
Kafka Broker metrics - Leader count
Kafka Broker metrics - Offline Partitions Count
Kafka Broker metrics - Replication Max Lag
Kafka Broker metrics - (AWS MSK) data_logs_disk_used
Producer metrics - is stuff working at all? - ruby_kafka.api.calls
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.messages
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.attempts
Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.errors
Consumer metrics - does stuff work at all? - ruby_kafka.api.calls
Consumer metrics - does stuff work at all? - ruby_kafka.api.errors
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.lag{*} by
{topic,partition}
Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.messages by
{topic,partition}
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.leave_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.sync_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.consumer.join_group
Consumer metrics - is there anything wrong going on with
consumers? - ruby_kafka.fetcher.queue_size
Thanks!