$30 off During Our Annual Pro Sale. View Details »

Kafka for Rubyists: Monitoring

Kafka for Rubyists: Monitoring

karol.galanciak

December 20, 2020
Tweet

More Decks by karol.galanciak

Other Decks in Programming

Transcript

  1. Kafka & Karafka - Monitoring

    View Slide

  2. Monitoring - Warning
    - No monitoring? Forget about using Kafka on
    production

    View Slide

  3. Monitoring - Kafka cluster itself vs. application-related metrics
    - Both are critical
    - Karafka uses ruby-kafka (versions 1.x) under
    the hood, which has a great integration with
    Datadog/Statsd

    View Slide

  4. Kafka Broker metrics
    - Under-replicated partitions

    View Slide

  5. Kafka Broker metrics
    - Active Controller Count

    View Slide

  6. Kafka Broker metrics
    - messages_in_per_sec

    View Slide

  7. Kafka Broker metrics
    - bytes_in_per_sec

    View Slide

  8. Kafka Broker metrics
    - bytes_out_per_sec

    View Slide

  9. Kafka Broker metrics
    - Leader count

    View Slide

  10. Kafka Broker metrics
    - Offline Partitions Count

    View Slide

  11. Kafka Broker metrics
    - Replication Max Lag

    View Slide

  12. Kafka Broker metrics
    - (AWS MSK) data_logs_disk_used

    View Slide

  13. Producer metrics - is stuff working at all?
    - ruby_kafka.api.calls

    View Slide

  14. Producer metrics - are messages getting published?
    - ruby_kafka.producer.deliver.messages

    View Slide

  15. Producer metrics - are messages getting published?
    - ruby_kafka.producer.deliver.attempts

    View Slide

  16. Producer metrics - are messages getting published?
    - ruby_kafka.producer.deliver.errors

    View Slide

  17. Consumer metrics - does stuff work at all?
    - ruby_kafka.api.calls

    View Slide

  18. Consumer metrics - does stuff work at all?
    - ruby_kafka.api.errors

    View Slide

  19. Consumer metrics - are messages getting consumed?
    - ruby_kafka.consumer.lag{*} by {topic,partition}

    View Slide

  20. Consumer metrics - are messages getting consumed?
    - ruby_kafka.consumer.messages by
    {topic,partition}

    View Slide

  21. Consumer metrics - is there anything wrong going on with consumers?
    - ruby_kafka.consumer.leave_group

    View Slide

  22. Consumer metrics - is there anything wrong going on with consumers?
    - ruby_kafka.consumer.sync_group

    View Slide

  23. Consumer metrics - is there anything wrong going on with consumers?
    - ruby_kafka.consumer.join_group

    View Slide

  24. Consumer metrics - is there anything wrong going on with consumers?
    - ruby_kafka.fetcher.queue_size

    View Slide

  25. Thanks!

    View Slide