Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ELK and Kafka

ELK and Kafka

These slides were presented at the Elasticsearch Meetup hosted at Linkedin: http://www.meetup.com/Silicon-Valley-Elastic-Fantastics/events/221527825/?eventId=221527825&action=detail

ABSTRACT: In this presentation, Tal covers how to scale Logstash using Kafka, including architecture and configuration. He also introduces the new Logstash Kafka plugins and how you can make the best use of them. Tal concludes the presentation by covering monitoring Kafka JMX reporter statistics using the ELK Stack, including a demo of on an ELK dashboard to collect and analyze Kafka JMX bean statistics.

Elastic Co

April 29, 2015
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. www.elastic.co 2 What Logstash Kafka and how Logstash can leverage

    it Message Serialization Monitoring Kafka with Elasticsearch, Logstash, Kibana
  2. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 3 Logstash …With LOTS of Plugins! Input Filter Output File JMX TCP Couchdb IRC Twitter Mutate Date Grok Ruby UserAgent Elasticsearch Postgres Nagios IRC Log Processing Pipeline
  3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 4 Logstash Config input { file { path => “/path/to/logs” } } filter { grok { match => { "message" => “%{IP:client}” } } } output { elasticsearch { protocol => “http” } }
  4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 6 Kafka … and a Logstash plugin! A High-throughput distributed messaging system
  5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 7 Input Plugin input { kafka { topic_id => “most_important”
 } }
  6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 8 Output Plugin output { kafka { topic_id => “most_important”
 } }
  7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 9 Custom Serializer One of the options in Kafka Producer output { kafka { … serializer.class => “kafka.serializer.StringEncoder” … } }
  8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 10 Custom Dots Serializer package org.logstash.dots; import kafka.serializer.Encoder; import kafka.utils.VerifiableProperties; public class DotsEncoder implements Encoder<String> { public DotsEncoder(VerifiableProperties props) { / /do nothing } @Override public byte[] toBytes(String s) { return ".".getBytes(); } } https:/ /github.com/talevy/kafka-dots-serializer
  9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 11 Custom Dots Serializer output { kafka { … serializer.class => “org.logstash.dots.DotsEncoder” … } }
  10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 12 Custom Serializer $ export CLASSPATH=“$CLASSPATH:/path/to/serializer.jar” $ bin/logstash -f /path/to/config In Logstash Environment logs in, …. out!
  11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 14 Kafka & JMX … Logstash has a plugin for that! Lots of metrics are exposed via JMX
  12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 15 JMX Plugin http://www.elastic.co/guide/en/logstash/current/plugins-inputs-jmx.html
  13. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 16 Monitoring Server Throughput { "host": "localhost", "port": 9999, "alias": "kafka.server", "queries": [ { "object_name": "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=*", "attributes": ["MeanRate"], "object_alias": "msg_per_sec-${topic}" } ] } jmx json configuration
  14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 17 Logstash Config input { jmx { path => “/path/to/jmxconf” polling_frequency => 5 } } output { elasticsearch { protocol => "http" } }
  15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 18 JMX Event { "@version" => "1", "@timestamp" => "2015-04-29T21:59:10.160Z", "host" => "localhost", "path" => “jmx-server", "type" => nil, "metric_path" => "kafka.server.msg_per_sec-topic2.MeanRate", "metric_value_number" => 1982.548103009822 }
  16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 20 Elasticsearch Index Management Daily Indices logstash-%{+YYYY.MM.dd} Curator A Python client for managing ES indices Important to know if you want to save disk space!
  17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 21 Monitoring Consumption export LS_JAVA_OPTS=“ -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=3000 -Dcom.sun.management.jmxremote.ssl=false" custom java opts for Logstash $ bin/logstash -e “input { kafka { topic_id => hello } }” running Logstash kafka consumer:
  18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written

    permission is strictly prohibited 22 Monitoring Consumer Lag { "host": "localhost", "port": 3001, "queries": [ { "object_name": "kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=*,partition=*" } ] } jmx json configuration