Life of a Logstash Event

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
March 19, 2015

Life of a Logstash Event

Join me for this presentation to follow the journey of a Logstash event as it travels through the internal Logstash processing pipeline. Understanding Logstash internals will help you make better design decisions for your streaming data processing architecture.

You will learn how data is transported in and out of Logstash through its input and output plugins, how Logstash decodes and encodes external data formats using the codec plugins and how the parsing, transformation and enrichment of this data is executed by the filter plugins.

We will follow an event as it travels through the pipeline stages and internal queuing, what happens when things start to break downstream, plus how congestion and backpressure is handled.

The Logstash pipeline is multithreaded, so you’ll also learn how to scale up and take advantage of all your host CPU cores and how to leverage parallelism to accelerate the processing throughput.

Logstash is an amazingly powerful and flexible tool. As a Logstash user, this presentation will help you optimize your logstash configuration and, for developers, you will have much better insights for contributing plugins.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

March 19, 2015
Tweet

Transcript

  1. Life of an Event in Logstash Colin Surprenant, Elasticsearch Software

    Engineer colin.surprenant@elastic.co colinsurprenant
  2. { } CC-BY-ND 4.0 Agenda 2 •Logstash quick intro •Pipeline

    overview •Plugin architecture •Scaling up Logstash •Pipeline stages •input •filter •output
  3. { } CC-BY-ND 4.0 Logstash quick intro Collect, parse and

    store logs 3
  4. { } CC-BY-ND 4.0 Logstash quick intro • Logs? 4

    ALL  THE  DATA  with  a  timestamp No  timestamp?  Hey!  
 we’ll  add  one  for  you.  
  5. { } CC-BY-ND 4.0 Logstash quick intro Transport & processing


    of
 streaming / continuous data 5
  6. { } CC-BY-ND 4.0 Logstash quick intro – Heterogenous formats

    and protocols – Unstructured format – Decentralized 6 Logging problems
  7. { } CC-BY-ND 4.0 Definitions • Event • Plugin (input,

    filter, output) • Pipeline 7
  8. { } CC-BY-ND 4.0 Logstash pipeline 8

  9. { } CC-BY-ND 4.0 Logstash pipeline 9 3 stages pipeline

  10. { } CC-BY-ND 4.0 Plugin architecture • ~200 plugins https://github.com/logstash-plugins

    • Input plugins: captures external data+format & transform it to logstash events • Filter plugins: process/transform events • Output plugins: send events to external destination & format 10
  11. { } CC-BY-ND 4.0 Plugin architecture • Codecs are plugins

    • Optional part of input and output plugins • encode and decode raw data that enter or exit the pipeline • Character encoding/transcoding into UTF-8 11 Codecs
  12. { } CC-BY-ND 4.0 Plugin architecture 12

  13. { } CC-BY-ND 4.0 Logstash pipeline 13 3 stages pipeline

  14. { } CC-BY-ND 4.0 Logstash pipeline 14 Internal queuing

  15. { } CC-BY-ND 4.0 Logstash pipeline 15 backpressure

  16. { } CC-BY-ND 4.0 Scaling out Logstash 16 shipper/indexer architecture

  17. { } CC-BY-ND 4.0 Scaling up the pipeline • Logstash

    pipeline is multithreaded • Each stage offers configurable concurrency options • Leverage multiple cores in host 17
  18. { } CC-BY-ND 4.0 Pipeline input stage 18

  19. { } CC-BY-ND 4.0 Pipeline input worker 19

  20. { } CC-BY-ND 4.0 Pipeline filter stage 20

  21. { } CC-BY-ND 4.0 Pipeline filter stage 21

  22. { } CC-BY-ND 4.0 Pipeline filter stage 22

  23. { } CC-BY-ND 4.0 Pipeline output stage 23

  24. { } CC-BY-ND 4.0 Pipeline output stage 24

  25. { } CC-BY-ND 4.0 Function compilation • Filter and output

    stages are compiled to a single function • Anyone knows or can guess why? 25
  26. { } CC-BY-ND 4.0 Function Compilation • Hint 26

  27. { } CC-BY-ND 4.0 Function Compilation • Conditionals • Message

    passing is not free 27
  28. { } CC-BY-ND 4.0 Logstash pipeline 28

  29. { } Thank you! Colin Surprenant, Elasticsearch Software Engineer colin.surprenant@elastic.co

    colinsurprenant
  30. { } This work is licensed under the Creative Commons

    Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0