Kafka for Rubyists: Monitoring

Transcript

1. Kafka & Karafka - Monitoring
2. Monitoring - Warning - No monitoring? Forget about using Kafka on production
3. Monitoring - Kafka cluster itself vs. application-related metrics - Both are critical - Karafka uses ruby-kafka (versions 1.x) under the hood, which has a great integration with Datadog/Statsd
4. Kafka Broker metrics - Under-replicated partitions
5. Kafka Broker metrics - Active Controller Count
6. Kafka Broker metrics - messages_in_per_sec
7. Kafka Broker metrics - bytes_in_per_sec
8. Kafka Broker metrics - bytes_out_per_sec
9. Kafka Broker metrics - Leader count
10. Kafka Broker metrics - Offline Partitions Count
11. Kafka Broker metrics - Replication Max Lag
12. Kafka Broker metrics - (AWS MSK) data_logs_disk_used
13. Producer metrics - is stuff working at all? - ruby_kafka.api.calls
14. Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.messages
15. Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.attempts
16. Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.errors
17. Consumer metrics - does stuff work at all? - ruby_kafka.api.calls
18. Consumer metrics - does stuff work at all? - ruby_kafka.api.errors
19. Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.lag{*} by {topic,partition}
20. Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.messages by {topic,partition}
21. Consumer metrics - is there anything wrong going on with consumers? - ruby_kafka.consumer.leave_group
22. Consumer metrics - is there anything wrong going on with consumers? - ruby_kafka.consumer.sync_group
23. Consumer metrics - is there anything wrong going on with consumers? - ruby_kafka.consumer.join_group
24. Consumer metrics - is there anything wrong going on with consumers? - ruby_kafka.fetcher.queue_size
25. Thanks!