Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's new in Logstash

What's new in Logstash

What's New in Logstash - Jordan Sissel and Suyog Rao

In this talk, we’ll give you an exciting preview into the world of Logstash 2.0, our next major release. We are focusing on three main themes for 2.0: resiliency, manageability, and performance improvements. We’ll take you through a tour of these themes and some use cases that we intend to address. We’ll talk about what a Logstash cluster will look like, plus APIs that will be used to interact with it. Under our resiliency project, we focus on data delivery guarantees, end-to-end acknowledgement and retry mechanisms. Finally, we’ll give you more detail into where the Logstash Forwarder is headed and performance improvements to make Logstash faster, so you can process more data in less time

Attendees will leave ready to plan a fabulous future with Logstash.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

March 10, 2015
Tweet

More Decks by Elastic Co

Other Decks in Programming

Transcript

  1. What’s new in Logstash @jordansissel @suyograo

  2. { } CC-BY-ND 4.0 What’s new in Logstash Today’s Queue

    2 Inputs Filters Outputs
  3. { } CC-BY-ND 4.0 What’s new in Logstash Today In-memory.

    Small, fixed size. Events are lost upon crashing. 3
  4. { } CC-BY-ND 4.0 What’s new in Logstash Future: Disk-backed

    Disk-backed. Small fixed size. No loss on crash. 4
  5. { } CC-BY-ND 4.0 What’s new in Logstash Future: Variable

    Size 5 … … … … … Disk-backed. Small fixed Variable size.
  6. { } CC-BY-ND 4.0 What’s new in Logstash Typical ELK

    Stack 6 elasticsearch Payments Server Database Web Server
  7. { } CC-BY-ND 4.0 What’s new in Logstash Simplified ELK

    Stack 7 elasticsearch Payments Server Database Web Server … …… ……
  8. { } CC-BY-ND 4.0 What’s new in Logstash Fault Management

    8 Filters Outputs … … … … … ❌
  9. { } CC-BY-ND 4.0 What’s new in Logstash Fault Management

    9 Filters Outputs … … … … … ❌ … …… …… Dead Letter Queue …to dead letter input
  10. { } CC-BY-ND 4.0 What’s new in Logstash Improve Resource

    Usage 10 … …… …… elasticsearch flush
  11. { } CC-BY-ND 4.0 What’s new in Logstash Improve Resource

    Usage 11 … …… …… elasticsearch flush
  12. Manageability Coming in Logstash 2.x

  13. { } CC-BY-ND 4.0 What’s new in Logstash Logstash configuration

    13 Web Server
  14. { } CC-BY-ND 4.0 What’s new in Logstash Config changes

    14
  15. { } CC-BY-ND 4.0 What’s new in Logstash Survey Question:

    How are you managing Logstash instances? 15
  16. { } CC-BY-ND 4.0 What’s new in Logstash Most users

    manage manually 16
  17. { } CC-BY-ND 4.0 What’s new in Logstash API driven

    changes 17 Dynamic changes, no need to restart instances
  18. { } CC-BY-ND 4.0 What’s new in Logstash Clustered Logstash

    18 Payments Server Database Web Server
  19. { } CC-BY-ND 4.0 What’s new in Logstash Clustering Implementation

    19 Payments Server Database Web Server elasticsearch
  20. { } CC-BY-ND 4.0 What’s new in Logstash Clustering benefits

    • Ease of control and management • Provides groundwork for future changes • “Tags” to mark instances — which means you could run multiple pipelines • High availability and load balancing 20
  21. { } CC-BY-ND 4.0 What’s new in Logstash Survey Question:

    What metrics would you like to see exposed 21
  22. { } CC-BY-ND 4.0 What’s new in Logstash Metrics 22

  23. { } CC-BY-ND 4.0 What’s new in Logstash Operational Visibility

    • “Why is grok being slow?” • “How many events are coming in over syslog?” • “What is the latency of events through Logstash?” 23
  24. { } CC-BY-ND 4.0 What’s new in Logstash Recap: Manageability

    • API Driven • No restarts for config changes • Support both single instance and clustered instance • Pluggable interface for storing state (Elasticsearch) • More metrics from Logstash 24
  25. Logstash Forwarder Future Experiments

  26. { } CC-BY-ND 4.0 What’s new in Logstash 26

  27. { } CC-BY-ND 4.0 What’s new in Logstash The Idea:

    Unify Codebase Hopes: Easier to maintain + fewer bugs Better community engagement Fewer moving parts 27 Fears: Possible performance loss Difficulty in packaging Larger resource usage What if logstash-forwarder was just logstash?
  28. { } CC-BY-ND 4.0 What’s new in Logstash The Experiment:

    Logstash under MRI 1. Make Logstash’s tests pass under MRI 2.2 2. Observe: file input to lumberjack output 3. Create a single package with no dependencies 28
  29. { } CC-BY-ND 4.0 What’s new in Logstash Side effects:

    Logstash under MRI 1. Fast start time 2. Improved first-time experience 3. More deployment options (MRI, JRuby) 29
  30. { } CC-BY-ND 4.0 What’s new in Logstash 30 FYI:

    Logstash is still best in JRuby.
  31. { } CC-BY-ND 4.0 What’s new in Logstash worst case:

    logstash-forwarder stays in Go and we invest in more heavily in it. 31
  32. { } CC-BY-ND 4.0 What’s new in Logstash Contribute via

    Github 32
  33. { } CC-BY-ND 4.0 What’s new in Logstash 33

  34. Thank You! @jordansissel and @suyograo

  35. { } CC-BY-ND 4.0 What’s new in Logstash Attribution &

    References Hard drive icon by Mario Verduzco from the noun project https://thenounproject.com/icon/30771/download-options/ Survey icon by Brennan Novak from the noun project http://thenounproject.com/term/survey/16392/ Terminal icon by useiconic.com from the noun project http://thenounproject.com/term/terminal/45367/ “Dr. Evil” image from “Austin Powers” movie. Image captioned by memegenerator.net 35
  36. { } This work is licensed under the Creative Commons

    Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0 What’s new in Logstash 36