How to scale a Logging Infrastructure

Logging infrastructure using ELK + Kafka

Transcript

1. How do you scale a logging infrastructure to accept a billion messages a day? Paul Stack http://twitter.com/stack72 mail: paul@paulstack.co.uk
2. About Me Infrastructure Engineer for a cool startup :) Reformed ASP.NET / C# Developer DevOps Extremist Conference Junkie
3. Background Project was to replace the legacy ‘logging solution’
4. Iteration 0: A Developer created a single box with the ELK all in 1 jar
5. Time to make it production ready now
7. Iteration 1: Using Redis as the input mechanism for LogStash
10. Enter Apache Kafka
11. “Kafka is a distributed publish- subscribe messaging system that is designed to be fast, scalable, and durable” Source: Cloudera Blog
12. Introduction to Kafka • Kafka is made up of ‘topics’, ‘producers’, ‘consumers’ and ‘brokers’ • Communication is via TCP • Backed by Zookeeper
13. Kafka Topics Source: http://kafka.apache.org/documentation.html
14. Kafka Producers • Producers are responsible to chose what topic to publish data to • The producer is responsible for choosing a partition to write to • Can be handled round robin or partition functions
15. Kafka Consumers • Consumption can be done via: • queuing • pub-sub
16. Kafka Consumers • Kafka consumer group • Strong ordering
17. Kafka Consumers • Strong ordering
18. https://github.com/opentable/puppet-exhibitor
20. Iteration 2 Introduction of Kafka
23. Iteration 3 Further ‘Improvements’ to the cluster layout
25. The Numbers • Logs kept in ES for 30 days then archived • 12 billion documents active in ES • ES space was about 25 - 30TB in EBS volumes • Average Doc Size ~ 1.2KB • V-Day 2015: ~750M docs collected without failure
26. What about metrics and monitoring?
27. Monitoring - Nagios • Alerts on • ES Cluster • zK and Kafka Nodes • Logstash / Redis nodes
29. https://github.com/stack72/nagios-elasticsearch
30. Metrics - Kafka Offset Monitor
31. https://github.com/opentable/KafkaOffsetMonitor
32. Metrics - ElasticSearch
36. Visibility Rocks!
38. So what would I do differently?
39. Questions?
40. Paul Stack @stack72