ELK and Kafka

These slides were presented at the Elasticsearch Meetup hosted at Linkedin: http://www.meetup.com/Silicon-Valley-Elastic-Fantastics/events/221527825/?eventId=221527825&action=detail

ABSTRACT: In this presentation, Tal covers how to scale Logstash using Kafka, including architecture and configuration. He also introduces the new Logstash Kafka plugins and how you can make the best use of them. Tal concludes the presentation by covering monitoring Kafka JMX reporter statistics using the ELK Stack, including a demo of on an ELK dashboard to collect and analyze Kafka JMX bean statistics.

Transcript

1. ELK and Kafka Tal levy April 29, 2015
2. www.elastic.co 2 What Logstash Kafka and how Logstash can leverage it Message Serialization Monitoring Kafka with Elasticsearch, Logstash, Kibana
3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 3 Logstash …With LOTS of Plugins! Input Filter Output File JMX TCP Couchdb IRC Twitter Mutate Date Grok Ruby UserAgent Elasticsearch Postgres Nagios IRC Log Processing Pipeline
4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 4 Logstash Config input { file { path => “/path/to/logs” } } filter { grok { match => { "message" => “%{IP:client}” } } } output { elasticsearch { protocol => “http” } }
5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 5 Where
6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 6 Kafka … and a Logstash plugin! A High-throughput distributed messaging system
7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 7 Input Plugin input { kafka { topic_id => “most_important”
 } }
8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 8 Output Plugin output { kafka { topic_id => “most_important”
 } }
9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 9 Custom Serializer One of the options in Kafka Producer output { kafka { … serializer.class => “kafka.serializer.StringEncoder” … } }
10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 10 Custom Dots Serializer package org.logstash.dots; import kafka.serializer.Encoder; import kafka.utils.VerifiableProperties; public class DotsEncoder implements Encoder<String> { public DotsEncoder(VerifiableProperties props) { / /do nothing } @Override public byte[] toBytes(String s) { return ".".getBytes(); } } https:/ /github.com/talevy/kafka-dots-serializer
11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 11 Custom Dots Serializer output { kafka { … serializer.class => “org.logstash.dots.DotsEncoder” … } }
12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 12 Custom Serializer $ export CLASSPATH=“$CLASSPATH:/path/to/serializer.jar” $ bin/logstash -f /path/to/config In Logstash Environment logs in, …. out!
13. Monitoring Kafka with ELK
14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 14 Kafka & JMX … Logstash has a plugin for that! Lots of metrics are exposed via JMX
15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 15 JMX Plugin http://www.elastic.co/guide/en/logstash/current/plugins-inputs-jmx.html
16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 16 Monitoring Server Throughput { "host": "localhost", "port": 9999, "alias": "kafka.server", "queries": [ { "object_name": "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=*", "attributes": ["MeanRate"], "object_alias": "msg_per_sec-${topic}" } ] } jmx json configuration
17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 17 Logstash Config input { jmx { path => “/path/to/jmxconf” polling_frequency => 5 } } output { elasticsearch { protocol => "http" } }
18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 18 JMX Event { "@version" => "1", "@timestamp" => "2015-04-29T21:59:10.160Z", "host" => "localhost", "path" => “jmx-server", "type" => nil, "metric_path" => "kafka.server.msg_per_sec-topic2.MeanRate", "metric_value_number" => 1982.548103009822 }
19. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 19 Throughput
20. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 20 Elasticsearch Index Management Daily Indices logstash-%{+YYYY.MM.dd} Curator A Python client for managing ES indices Important to know if you want to save disk space!
21. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 21 Monitoring Consumption export LS_JAVA_OPTS=“ -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=3000 -Dcom.sun.management.jmxremote.ssl=false" custom java opts for Logstash $ bin/logstash -e “input { kafka { topic_id => hello } }” running Logstash kafka consumer:
22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 22 Monitoring Consumer Lag { "host": "localhost", "port": 3001, "queries": [ { "object_name": "kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=*,partition=*" } ] } jmx json configuration
23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 23 Consumer Lag
24. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 24 Together
25. Thank You!