Life of a Logstash Event

Join me for this presentation to follow the journey of a Logstash event as it travels through the internal Logstash processing pipeline. Understanding Logstash internals will help you make better design decisions for your streaming data processing architecture.

You will learn how data is transported in and out of Logstash through its input and output plugins, how Logstash decodes and encodes external data formats using the codec plugins and how the parsing, transformation and enrichment of this data is executed by the filter plugins.

We will follow an event as it travels through the pipeline stages and internal queuing, what happens when things start to break downstream, plus how congestion and backpressure is handled.

The Logstash pipeline is multithreaded, so you’ll also learn how to scale up and take advantage of all your host CPU cores and how to leverage parallelism to accelerate the processing throughput.

Logstash is an amazingly powerful and flexible tool. As a Logstash user, this presentation will help you optimize your logstash configuration and, for developers, you will have much better insights for contributing plugins.

Transcript

1. Life of an Event in Logstash
   Colin Surprenant, Elasticsearch Software Engineer
   colin.surprenant@elastic.co
   colinsurprenant
   

   View Slide

2. { } CC-BY-ND 4.0
   Agenda
   2
   •Logstash quick intro
   •Pipeline overview
   •Plugin architecture
   •Scaling up Logstash
   •Pipeline stages
   •input
   •filter
   •output
   

   View Slide

3. { } CC-BY-ND 4.0
   Logstash quick intro
   Collect, parse and store logs
   3
   

   View Slide

4. { } CC-BY-ND 4.0
   Logstash quick intro
   • Logs?
   4
   ALL  THE  DATA  with  a  timestamp
   No  timestamp?  Hey!  

   we’ll  add  one  for  you.  
   

   View Slide

5. { } CC-BY-ND 4.0
   Logstash quick intro
   Transport & processing

   of

   streaming / continuous data
   5
   

   View Slide

6. { } CC-BY-ND 4.0
   Logstash quick intro
   – Heterogenous formats and protocols
   – Unstructured format
   – Decentralized
   6
   Logging problems
   

   View Slide

7. { } CC-BY-ND 4.0
   Definitions
   • Event
   • Plugin (input, filter, output)
   • Pipeline
   7
   

   View Slide

8. { } CC-BY-ND 4.0
   Logstash pipeline
   8
   

   View Slide

9. { } CC-BY-ND 4.0
   Logstash pipeline
   9
   3 stages pipeline
   

   View Slide

10. { } CC-BY-ND 4.0
    Plugin architecture
    • ~200 plugins https://github.com/logstash-plugins
    • Input plugins: captures external data+format & transform
    it to logstash events
    • Filter plugins: process/transform events
    • Output plugins: send events to external destination &
    format
    10
    

    View Slide

11. { } CC-BY-ND 4.0
    Plugin architecture
    • Codecs are plugins
    • Optional part of input and output plugins
    • encode and decode raw data that enter or exit the
    pipeline
    • Character encoding/transcoding into UTF-8
    11
    Codecs
    

    View Slide

12. { } CC-BY-ND 4.0
    Plugin architecture
    12
    

    View Slide

13. { } CC-BY-ND 4.0
    Logstash pipeline
    13
    3 stages pipeline
    

    View Slide

14. { } CC-BY-ND 4.0
    Logstash pipeline
    14
    Internal queuing
    

    View Slide

15. { } CC-BY-ND 4.0
    Logstash pipeline
    15
    backpressure
    

    View Slide

16. { } CC-BY-ND 4.0
    Scaling out Logstash
    16
    shipper/indexer architecture
    

    View Slide

17. { } CC-BY-ND 4.0
    Scaling up the pipeline
    • Logstash pipeline is multithreaded
    • Each stage offers configurable concurrency options
    • Leverage multiple cores in host
    17
    

    View Slide

18. { } CC-BY-ND 4.0
    Pipeline input stage
    18
    

    View Slide

19. { } CC-BY-ND 4.0
    Pipeline input worker
    19
    

    View Slide

20. { } CC-BY-ND 4.0
    Pipeline filter stage
    20
    

    View Slide

21. { } CC-BY-ND 4.0
    Pipeline filter stage
    21
    

    View Slide

22. { } CC-BY-ND 4.0
    Pipeline filter stage
    22
    

    View Slide

23. { } CC-BY-ND 4.0
    Pipeline output stage
    23
    

    View Slide

24. { } CC-BY-ND 4.0
    Pipeline output stage
    24
    

    View Slide

25. { } CC-BY-ND 4.0
    Function compilation
    • Filter and output stages are compiled to a single
    function
    • Anyone knows or can guess why?
    25
    

    View Slide

26. { } CC-BY-ND 4.0
    Function Compilation
    • Hint
    26
    

    View Slide

27. { } CC-BY-ND 4.0
    Function Compilation
    • Conditionals
    • Message passing is not free
    27
    

    View Slide

28. { } CC-BY-ND 4.0
    Logstash pipeline
    28
    

    View Slide

29. { }
    Thank you!
    Colin Surprenant, Elasticsearch Software Engineer
    colin.surprenant@elastic.co
    colinsurprenant
    

    View Slide

30. { }
    This work is licensed under the Creative Commons
    Attribution-NoDerivatives 4.0 International License.
    To view a copy of this license, visit:
    http://creativecommons.org/licenses/by-nd/4.0/
    or send a letter to:
    Creative Commons
    PO Box 1866
    Mountain View, CA 94042
    USA
    CC-BY-ND 4.0
    

    View Slide