Kafka for Rubyists: Monitoring

Transcript

1. Kafka & Karafka - Monitoring
   

   View Slide

2. Monitoring - Warning
   - No monitoring? Forget about using Kafka on
   production
   

   View Slide

3. Monitoring - Kafka cluster itself vs. application-related metrics
   - Both are critical
   - Karafka uses ruby-kafka (versions 1.x) under
   the hood, which has a great integration with
   Datadog/Statsd
   

   View Slide

4. Kafka Broker metrics
   - Under-replicated partitions
   

   View Slide

5. Kafka Broker metrics
   - Active Controller Count
   

   View Slide

6. Kafka Broker metrics
   - messages_in_per_sec
   

   View Slide

7. Kafka Broker metrics
   - bytes_in_per_sec
   

   View Slide

8. Kafka Broker metrics
   - bytes_out_per_sec
   

   View Slide

9. Kafka Broker metrics
   - Leader count
   

   View Slide

10. Kafka Broker metrics
    - Offline Partitions Count
    

    View Slide

11. Kafka Broker metrics
    - Replication Max Lag
    

    View Slide

12. Kafka Broker metrics
    - (AWS MSK) data_logs_disk_used
    

    View Slide

13. Producer metrics - is stuff working at all?
    - ruby_kafka.api.calls
    

    View Slide

14. Producer metrics - are messages getting published?
    - ruby_kafka.producer.deliver.messages
    

    View Slide

15. Producer metrics - are messages getting published?
    - ruby_kafka.producer.deliver.attempts
    

    View Slide

16. Producer metrics - are messages getting published?
    - ruby_kafka.producer.deliver.errors
    

    View Slide

17. Consumer metrics - does stuff work at all?
    - ruby_kafka.api.calls
    

    View Slide

18. Consumer metrics - does stuff work at all?
    - ruby_kafka.api.errors
    

    View Slide

19. Consumer metrics - are messages getting consumed?
    - ruby_kafka.consumer.lag{*} by {topic,partition}
    

    View Slide

20. Consumer metrics - are messages getting consumed?
    - ruby_kafka.consumer.messages by
    {topic,partition}
    

    View Slide

21. Consumer metrics - is there anything wrong going on with consumers?
    - ruby_kafka.consumer.leave_group
    

    View Slide

22. Consumer metrics - is there anything wrong going on with consumers?
    - ruby_kafka.consumer.sync_group
    

    View Slide

23. Consumer metrics - is there anything wrong going on with consumers?
    - ruby_kafka.consumer.join_group
    

    View Slide

24. Consumer metrics - is there anything wrong going on with consumers?
    - ruby_kafka.fetcher.queue_size
    

    View Slide

25. Thanks!
    

    View Slide