Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka for Rubyists: Monitoring

Kafka for Rubyists: Monitoring

347a1b64f3a6e38981bc99b53919e2b1?s=128

karol.galanciak

December 20, 2020
Tweet

Transcript

  1. Kafka & Karafka - Monitoring

  2. Monitoring - Warning - No monitoring? Forget about using Kafka

    on production
  3. Monitoring - Kafka cluster itself vs. application-related metrics - Both

    are critical - Karafka uses ruby-kafka (versions 1.x) under the hood, which has a great integration with Datadog/Statsd
  4. Kafka Broker metrics - Under-replicated partitions

  5. Kafka Broker metrics - Active Controller Count

  6. Kafka Broker metrics - messages_in_per_sec

  7. Kafka Broker metrics - bytes_in_per_sec

  8. Kafka Broker metrics - bytes_out_per_sec

  9. Kafka Broker metrics - Leader count

  10. Kafka Broker metrics - Offline Partitions Count

  11. Kafka Broker metrics - Replication Max Lag

  12. Kafka Broker metrics - (AWS MSK) data_logs_disk_used

  13. Producer metrics - is stuff working at all? - ruby_kafka.api.calls

  14. Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.messages

  15. Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.attempts

  16. Producer metrics - are messages getting published? - ruby_kafka.producer.deliver.errors

  17. Consumer metrics - does stuff work at all? - ruby_kafka.api.calls

  18. Consumer metrics - does stuff work at all? - ruby_kafka.api.errors

  19. Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.lag{*} by

    {topic,partition}
  20. Consumer metrics - are messages getting consumed? - ruby_kafka.consumer.messages by

    {topic,partition}
  21. Consumer metrics - is there anything wrong going on with

    consumers? - ruby_kafka.consumer.leave_group
  22. Consumer metrics - is there anything wrong going on with

    consumers? - ruby_kafka.consumer.sync_group
  23. Consumer metrics - is there anything wrong going on with

    consumers? - ruby_kafka.consumer.join_group
  24. Consumer metrics - is there anything wrong going on with

    consumers? - ruby_kafka.fetcher.queue_size
  25. Thanks!