What's new in Logstash

What's New in Logstash - Jordan Sissel and Suyog Rao

In this talk, we’ll give you an exciting preview into the world of Logstash 2.0, our next major release. We are focusing on three main themes for 2.0: resiliency, manageability, and performance improvements. We’ll take you through a tour of these themes and some use cases that we intend to address. We’ll talk about what a Logstash cluster will look like, plus APIs that will be used to interact with it. Under our resiliency project, we focus on data delivery guarantees, end-to-end acknowledgement and retry mechanisms. Finally, we’ll give you more detail into where the Logstash Forwarder is headed and performance improvements to make Logstash faster, so you can process more data in less time

Attendees will leave ready to plan a fabulous future with Logstash.

Transcript

1. What’s new in Logstash @jordansissel @suyograo
2. { } CC-BY-ND 4.0 What’s new in Logstash Today’s Queue 2 Inputs Filters Outputs
3. { } CC-BY-ND 4.0 What’s new in Logstash Today In-memory. Small, fixed size. Events are lost upon crashing. 3
4. { } CC-BY-ND 4.0 What’s new in Logstash Future: Disk-backed Disk-backed. Small fixed size. No loss on crash. 4
5. { } CC-BY-ND 4.0 What’s new in Logstash Future: Variable Size 5 … … … … … Disk-backed. Small fixed Variable size.
6. { } CC-BY-ND 4.0 What’s new in Logstash Typical ELK Stack 6 elasticsearch Payments Server Database Web Server
7. { } CC-BY-ND 4.0 What’s new in Logstash Simplified ELK Stack 7 elasticsearch Payments Server Database Web Server … …… ……
8. { } CC-BY-ND 4.0 What’s new in Logstash Fault Management 8 Filters Outputs … … … … … ❌
9. { } CC-BY-ND 4.0 What’s new in Logstash Fault Management 9 Filters Outputs … … … … … ❌ … …… …… Dead Letter Queue …to dead letter input
10. { } CC-BY-ND 4.0 What’s new in Logstash Improve Resource Usage 10 … …… …… elasticsearch flush
11. { } CC-BY-ND 4.0 What’s new in Logstash Improve Resource Usage 11 … …… …… elasticsearch flush
12. Manageability Coming in Logstash 2.x
13. { } CC-BY-ND 4.0 What’s new in Logstash Logstash configuration 13 Web Server
14. { } CC-BY-ND 4.0 What’s new in Logstash Config changes 14
15. { } CC-BY-ND 4.0 What’s new in Logstash Survey Question: How are you managing Logstash instances? 15
16. { } CC-BY-ND 4.0 What’s new in Logstash Most users manage manually 16
17. { } CC-BY-ND 4.0 What’s new in Logstash API driven changes 17 Dynamic changes, no need to restart instances
18. { } CC-BY-ND 4.0 What’s new in Logstash Clustered Logstash 18 Payments Server Database Web Server
19. { } CC-BY-ND 4.0 What’s new in Logstash Clustering Implementation 19 Payments Server Database Web Server elasticsearch
20. { } CC-BY-ND 4.0 What’s new in Logstash Clustering benefits • Ease of control and management • Provides groundwork for future changes • “Tags” to mark instances — which means you could run multiple pipelines • High availability and load balancing 20
21. { } CC-BY-ND 4.0 What’s new in Logstash Survey Question: What metrics would you like to see exposed 21
22. { } CC-BY-ND 4.0 What’s new in Logstash Metrics 22
23. { } CC-BY-ND 4.0 What’s new in Logstash Operational Visibility • “Why is grok being slow?” • “How many events are coming in over syslog?” • “What is the latency of events through Logstash?” 23
24. { } CC-BY-ND 4.0 What’s new in Logstash Recap: Manageability • API Driven • No restarts for config changes • Support both single instance and clustered instance • Pluggable interface for storing state (Elasticsearch) • More metrics from Logstash 24
25. Logstash Forwarder Future Experiments
26. { } CC-BY-ND 4.0 What’s new in Logstash 26
27. { } CC-BY-ND 4.0 What’s new in Logstash The Idea: Unify Codebase Hopes: Easier to maintain + fewer bugs Better community engagement Fewer moving parts 27 Fears: Possible performance loss Difficulty in packaging Larger resource usage What if logstash-forwarder was just logstash?
28. { } CC-BY-ND 4.0 What’s new in Logstash The Experiment: Logstash under MRI 1. Make Logstash’s tests pass under MRI 2.2 2. Observe: file input to lumberjack output 3. Create a single package with no dependencies 28
29. { } CC-BY-ND 4.0 What’s new in Logstash Side effects: Logstash under MRI 1. Fast start time 2. Improved first-time experience 3. More deployment options (MRI, JRuby) 29
30. { } CC-BY-ND 4.0 What’s new in Logstash 30 FYI: Logstash is still best in JRuby.
31. { } CC-BY-ND 4.0 What’s new in Logstash worst case: logstash-forwarder stays in Go and we invest in more heavily in it. 31
32. { } CC-BY-ND 4.0 What’s new in Logstash Contribute via Github 32
33. { } CC-BY-ND 4.0 What’s new in Logstash 33
34. Thank You! @jordansissel and @suyograo
35. { } CC-BY-ND 4.0 What’s new in Logstash Attribution & References Hard drive icon by Mario Verduzco from the noun project https://thenounproject.com/icon/30771/download-options/ Survey icon by Brennan Novak from the noun project http://thenounproject.com/term/survey/16392/ Terminal icon by useiconic.com from the noun project http://thenounproject.com/term/terminal/45367/ “Dr. Evil” image from “Austin Powers” movie. Image captioned by memegenerator.net 35
36. { } This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0 What’s new in Logstash 36