Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Life of a Logstash Event

Life of a Logstash Event

Join me for this presentation to follow the journey of a Logstash event as it travels through the internal Logstash processing pipeline. Understanding Logstash internals will help you make better design decisions for your streaming data processing architecture.

You will learn how data is transported in and out of Logstash through its input and output plugins, how Logstash decodes and encodes external data formats using the codec plugins and how the parsing, transformation and enrichment of this data is executed by the filter plugins.

We will follow an event as it travels through the pipeline stages and internal queuing, what happens when things start to break downstream, plus how congestion and backpressure is handled.

The Logstash pipeline is multithreaded, so you’ll also learn how to scale up and take advantage of all your host CPU cores and how to leverage parallelism to accelerate the processing throughput.

Logstash is an amazingly powerful and flexible tool. As a Logstash user, this presentation will help you optimize your logstash configuration and, for developers, you will have much better insights for contributing plugins.

Elastic Co

March 19, 2015

More Decks by Elastic Co

Other Decks in Technology


  1. { } CC-BY-ND 4.0 Agenda 2 •Logstash quick intro •Pipeline

    overview •Plugin architecture •Scaling up Logstash •Pipeline stages •input •filter •output
  2. { } CC-BY-ND 4.0 Logstash quick intro • Logs? 4

    ALL  THE  DATA  with  a  timestamp No  timestamp?  Hey!  
 we’ll  add  one  for  you.  
  3. { } CC-BY-ND 4.0 Logstash quick intro – Heterogenous formats

    and protocols – Unstructured format – Decentralized 6 Logging problems
  4. { } CC-BY-ND 4.0 Plugin architecture • ~200 plugins https://github.com/logstash-plugins

    • Input plugins: captures external data+format & transform it to logstash events • Filter plugins: process/transform events • Output plugins: send events to external destination & format 10
  5. { } CC-BY-ND 4.0 Plugin architecture • Codecs are plugins

    • Optional part of input and output plugins • encode and decode raw data that enter or exit the pipeline • Character encoding/transcoding into UTF-8 11 Codecs
  6. { } CC-BY-ND 4.0 Scaling up the pipeline • Logstash

    pipeline is multithreaded • Each stage offers configurable concurrency options • Leverage multiple cores in host 17
  7. { } CC-BY-ND 4.0 Function compilation • Filter and output

    stages are compiled to a single function • Anyone knows or can guess why? 25
  8. { } This work is licensed under the Creative Commons

    Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0