Elasticon_-_What_s_new_in_Logstash.pdf

Transcript

1. What’s new in Logstash @jordansissel @suyograo
2. { } CC-BY-ND 4.0 What’s new in Logstash Today’s Queue 2 Inputs Filters Outputs
3. { } CC-BY-ND 4.0 What’s new in Logstash Today In-memory. Small, fixed size. Events are lost upon crashing. 3
4. { } CC-BY-ND 4.0 What’s new in Logstash Future: Disk-backed Disk-backed. Small fixed size. No loss on crash. 4
5. { } CC-BY-ND 4.0 What’s new in Logstash Future: Variable Size 5 … … … … … Disk-backed. Small fixed Variable size.
6. { } CC-BY-ND 4.0 What’s new in Logstash Typical ELK Stack 6 elasticsearch Payments Server Database Web Server
7. { } CC-BY-ND 4.0 What’s new in Logstash Simplified ELK Stack 7 elasticsearch Payments Server Database Web Server … …… ……
8. { } CC-BY-ND 4.0 What’s new in Logstash Fault Management 8 Filters Outputs … … … … … ❌
9. { } CC-BY-ND 4.0 What’s new in Logstash Fault Management 9 Filters Outputs … … … … … ❌ … …… …… Dead Letter Queue …to dead letter input
10. { } CC-BY-ND 4.0 What’s new in Logstash Improve Resource Usage 10 … …… …… elasticsearch flush
11. { } CC-BY-ND 4.0 What’s new in Logstash Improve Resource Usage 11 … …… …… elasticsearch flush
12. Manageability Coming in Logstash 2.x
13. { } CC-BY-ND 4.0 What’s new in Logstash Logstash configuration 13 Web Server
14. { } CC-BY-ND 4.0 What’s new in Logstash Config changes 14
15. { } CC-BY-ND 4.0 What’s new in Logstash Survey Question: How are you managing Logstash instances? 15
16. { } CC-BY-ND 4.0 What’s new in Logstash Most users manage manually 16
17. { } CC-BY-ND 4.0 What’s new in Logstash API driven changes 17 Dynamic changes, no need to restart instances
18. { } CC-BY-ND 4.0 What’s new in Logstash Clustered Logstash 18 Payments Server Database Web Server
19. { } CC-BY-ND 4.0 What’s new in Logstash Clustering Implementation 19 Payments Server Database Web Server elasticsearch
20. { } CC-BY-ND 4.0 What’s new in Logstash Clustering benefits • Ease of control and management • Provides groundwork for future changes • “Tags” to mark instances — which means you could run multiple pipelines • High availability and load balancing 20
21. { } CC-BY-ND 4.0 What’s new in Logstash Survey Question: What metrics would you like to see exposed 21
22. { } CC-BY-ND 4.0 What’s new in Logstash Metrics 22
23. { } CC-BY-ND 4.0 What’s new in Logstash Operational Visibility • “Why is grok being slow?” • “How many events are coming in over syslog?” • “What is the latency of events through Logstash?” 23
24. { } CC-BY-ND 4.0 What’s new in Logstash Recap: Manageability • API Driven • No restarts for config changes • Support both single instance and clustered instance • Pluggable interface for storing state (Elasticsearch) • More metrics from Logstash 24
25. Logstash Forwarder Future Experiments
26. { } CC-BY-ND 4.0 What’s new in Logstash 26
27. { } CC-BY-ND 4.0 What’s new in Logstash The Idea: Unify Codebase Hopes: Easier to maintain + fewer bugs Better community engagement Fewer moving parts 27 Fears: Possible performance loss Difficulty in packaging Larger resource usage What if logstash-forwarder was just logstash?
28. { } CC-BY-ND 4.0 What’s new in Logstash The Experiment: Logstash under MRI 1. Make Logstash’s tests pass under MRI 2.2 2. Observe: file input to lumberjack output 3. Create a single package with no dependencies 28
29. { } CC-BY-ND 4.0 What’s new in Logstash Side effects: Logstash under MRI 1. Fast start time 2. Improved first-time experience 3. More deployment options (MRI, JRuby) 29
30. { } CC-BY-ND 4.0 What’s new in Logstash 30 FYI: Logstash is still best in JRuby.
31. { } CC-BY-ND 4.0 What’s new in Logstash worst case: logstash-forwarder stays in Go and we invest in more heavily in it. 31
32. { } CC-BY-ND 4.0 What’s new in Logstash Contribute via Github 32
33. { } CC-BY-ND 4.0 What’s new in Logstash 33
34. Thank You! @jordansissel and @suyograo
35. { } CC-BY-ND 4.0 What’s new in Logstash Attribution & References Hard drive icon by Mario Verduzco from the noun project https://thenounproject.com/icon/30771/download-options/ Survey icon by Brennan Novak from the noun project http://thenounproject.com/term/survey/16392/ Terminal icon by useiconic.com from the noun project http://thenounproject.com/term/terminal/45367/ “Dr. Evil” image from “Austin Powers” movie. Image captioned by memegenerator.net 35
36. { } This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0 What’s new in Logstash 36