Maximizing Logstash Performance

Presentation at OpenWest 2017, accompanied by live demonstrations at the command-line and in Kibana.

1. Optimizing Logstash Performance There's more than one way to parse a log...
2. Define the problem • Logstash is perceived as slow • No insight into performance bottlenecks • No idea how to gain insight
3. Logstash? Data In Data Out
4. Have no fear! Let's shine a light in there
5. Rule #1
6. –Jordan Sissel, creator of Logstash...and grok. “If you have to use grok, you've already lost.”
7. Toolset 1 • generator input plugin • dots codec • pipe viewer (pv)
8. Generator input { generator { lines => [ 'line1', 'line2', ... 'lineN', ] count => 123456 } }
9. dots codec output { stdout { codec => dots } }
10. pv -r, --rate show data transfer rate counter -W, --wait display nothing until first byte transferred $ bin/logstash -f mytest.conf | pv -Wr > /dev/null [42.0KiB/s]
11. –No one, Ever “I like waiting!”
12. Toolset 2 • Elasticsearch, Logstash, & Kibana (+ X-Pack) • Configure Logstash to send monitoring data • View in Kibana
13. X-Pack $ bin/elasticsearch-plugin install x-pack $ bin/kibana-plugin install x-pack $ bin/logstash-plugin install x-pack
14. Configure Kibana vi config/kibana.yml # If your Elasticsearch is protected with basic authentication, # these settings provide the username and password that the # Kibana server uses to perform maintenance on the Kibana # index at startup. Your Kibana users still need to # authenticate with Elasticsearch, which # is proxied through the Kibana server. elasticsearch.username: "elastic" elasticsearch.password: "changeme"
15. Configure Logstash vi config/logstash.yml # Periodically check if the configuration has changed and # reload the pipeline # This can also be triggered manually through the SIGHUP signal # config.reload.automatic: true xpack.monitoring.elasticsearch.url: "http://localhost:9200" xpack.monitoring.elasticsearch.username: elastic xpack.monitoring.elasticsearch.password: changeme
16. Ready for Launch • Start Elasticsearch • Start Kibana • Navigate to Monitoring page
17. Logstash is a Pipeline In case I neglected to mention it
18. Pipeline Truths • At most, Logstash can only move data as fast as it comes in • Unless dropped or eliminated by conditional, each event will exit each output. • If a filter or output plugin is slow or blocked, the entire pipeline will back up • Filters will slow the pipeline–Some a little, some a lot. • Logstash can only ship data as fast as the slowest output. • No, really. Not kidding.
19. Consider the following... output { plugin1 {...} plugin2 {...} plugin3 {...} }
20. Improving performance • Use brokers • Parallel pipelines • Staged pipelines
21. Parallel pipeline example input {...} filter {# NONE} output { redis {...} } redis input { redis {...} } filter { # all } output { plugin1 {...} } input { redis {...} } filter { # all } output { plugin1 {...} } input { redis {...} } filter { # all } output { plugin1 {...} }
22. Staged pipeline example output { elasticsearch {...} redis {...} } ES redis input { redis {...} } output { slow_output {...} }
23. Other potential bottlenecks • Persistent Queues • Conditionals • Especially if you're doing regular expressions in them
24. Future methods • Multiple pipelines from 1 JVM • Definable in logstash.yml • Each with auto-reload • Pipeline viewer (may be only in X-Pack at release) • See throughput not just as a sum of input/output, but at each plugin and conditional.
25. Conclusion • If you can't measure it, you can't improve it, so monitor it. A lot. • Use grok and regular expressions... • ...as sparingly as possible • Don't put all of your pipeline eggs in one basket... • ...unless you've measured it and it meets your expectations • Parallelize and stage your pipeline with brokers FTW
26. Resources • https://www.elastic.co/guide/en/logstash/current/performance- troubleshooting.html • https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html