Kafka for Rubyists: Monitoring

Transcript

1. Kafka & Karafka - Monitoring

2. Monitoring - Warning - No monitoring? Forget about using Kafka

   on production

3. Monitoring - Kafka cluster itself vs. application-related metrics - Both

   are critical - Karafka uses ruby-kafka (versions 1.x) under the hood, which has a great integration with Datadog/Statsd

4. Kafka Broker metrics - Under-replicated partitions

5. Kafka Broker metrics - Active Controller Count

6. Kafka Broker metrics - messages_in_per_sec

7. Kafka Broker metrics - bytes_in_per_sec

8. Kafka Broker metrics - bytes_out_per_sec

9. Kafka Broker metrics - Leader count

10. Kafka Broker metrics - Offline Partitions Count

11. Kafka Broker metrics - Replication Max Lag

12. Kafka Broker metrics - (AWS MSK) data_logs_disk_used

13. Producer metrics - is stuff working at all? - ruby_kafka.api.calls

14. Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.messages

15. Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.attempts

16. Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.errors

17. Consumer metrics - does stuff work at all? - ruby_kafka.api.calls

18. Consumer metrics - does stuff work at all? - ruby_kafka.api.errors

19. Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.lag{*} by

    {topic,partition}

20. Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.messages by

    {topic,partition}

21. Consumer metrics - is there anything wrong going on with

    consumers? - ruby_kafka.consumer.leave_group

22. Consumer metrics - is there anything wrong going on with

    consumers? - ruby_kafka.consumer.sync_group

23. Consumer metrics - is there anything wrong going on with

    consumers? - ruby_kafka.consumer.join_group

24. Consumer metrics - is there anything wrong going on with

    consumers? - ruby_kafka.fetcher.queue_size

25. Thanks!