Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticon_-_What_s_new_in_Logstash.pdf

Elastic Co
March 18, 2015
130

 Elasticon_-_What_s_new_in_Logstash.pdf

Elastic Co

March 18, 2015
Tweet

Transcript

  1. { } CC-BY-ND 4.0 What’s new in Logstash Today In-memory.

    Small, fixed size. Events are lost upon crashing. 3
  2. { } CC-BY-ND 4.0 What’s new in Logstash Future: Disk-backed

    Disk-backed. Small fixed size. No loss on crash. 4
  3. { } CC-BY-ND 4.0 What’s new in Logstash Future: Variable

    Size 5 … … … … … Disk-backed. Small fixed Variable size.
  4. { } CC-BY-ND 4.0 What’s new in Logstash Typical ELK

    Stack 6 elasticsearch Payments Server Database Web Server
  5. { } CC-BY-ND 4.0 What’s new in Logstash Simplified ELK

    Stack 7 elasticsearch Payments Server Database Web Server … …… ……
  6. { } CC-BY-ND 4.0 What’s new in Logstash Fault Management

    8 Filters Outputs … … … … … ❌
  7. { } CC-BY-ND 4.0 What’s new in Logstash Fault Management

    9 Filters Outputs … … … … … ❌ … …… …… Dead Letter Queue …to dead letter input
  8. { } CC-BY-ND 4.0 What’s new in Logstash Improve Resource

    Usage 10 … …… …… elasticsearch flush
  9. { } CC-BY-ND 4.0 What’s new in Logstash Improve Resource

    Usage 11 … …… …… elasticsearch flush
  10. { } CC-BY-ND 4.0 What’s new in Logstash Survey Question:

    How are you managing Logstash instances? 15
  11. { } CC-BY-ND 4.0 What’s new in Logstash API driven

    changes 17 Dynamic changes, no need to restart instances
  12. { } CC-BY-ND 4.0 What’s new in Logstash Clustered Logstash

    18 Payments Server Database Web Server
  13. { } CC-BY-ND 4.0 What’s new in Logstash Clustering Implementation

    19 Payments Server Database Web Server elasticsearch
  14. { } CC-BY-ND 4.0 What’s new in Logstash Clustering benefits

    • Ease of control and management • Provides groundwork for future changes • “Tags” to mark instances — which means you could run multiple pipelines • High availability and load balancing 20
  15. { } CC-BY-ND 4.0 What’s new in Logstash Survey Question:

    What metrics would you like to see exposed 21
  16. { } CC-BY-ND 4.0 What’s new in Logstash Operational Visibility

    • “Why is grok being slow?” • “How many events are coming in over syslog?” • “What is the latency of events through Logstash?” 23
  17. { } CC-BY-ND 4.0 What’s new in Logstash Recap: Manageability

    • API Driven • No restarts for config changes • Support both single instance and clustered instance • Pluggable interface for storing state (Elasticsearch) • More metrics from Logstash 24
  18. { } CC-BY-ND 4.0 What’s new in Logstash The Idea:

    Unify Codebase Hopes: Easier to maintain + fewer bugs Better community engagement Fewer moving parts 27 Fears: Possible performance loss Difficulty in packaging Larger resource usage What if logstash-forwarder was just logstash?
  19. { } CC-BY-ND 4.0 What’s new in Logstash The Experiment:

    Logstash under MRI 1. Make Logstash’s tests pass under MRI 2.2 2. Observe: file input to lumberjack output 3. Create a single package with no dependencies 28
  20. { } CC-BY-ND 4.0 What’s new in Logstash Side effects:

    Logstash under MRI 1. Fast start time 2. Improved first-time experience 3. More deployment options (MRI, JRuby) 29
  21. { } CC-BY-ND 4.0 What’s new in Logstash 30 FYI:

    Logstash is still best in JRuby.
  22. { } CC-BY-ND 4.0 What’s new in Logstash worst case:

    logstash-forwarder stays in Go and we invest in more heavily in it. 31
  23. { } CC-BY-ND 4.0 What’s new in Logstash Attribution &

    References Hard drive icon by Mario Verduzco from the noun project https://thenounproject.com/icon/30771/download-options/ Survey icon by Brennan Novak from the noun project http://thenounproject.com/term/survey/16392/ Terminal icon by useiconic.com from the noun project http://thenounproject.com/term/terminal/45367/ “Dr. Evil” image from “Austin Powers” movie. Image captioned by memegenerator.net 35
  24. { } This work is licensed under the Creative Commons

    Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0 What’s new in Logstash 36