Tweet
Tweet

Slide 1

Slide 1 text

Life of an Event in Logstash Colin Surprenant, Elasticsearch Software Engineer colin.surprenant@elastic.co colinsurprenant

Slide 2

Slide 2 text

{ } CC-BY-ND 4.0 Agenda 2 •Logstash quick intro •Pipeline overview •Plugin architecture •Scaling up Logstash •Pipeline stages •input •filter •output

Slide 3

Slide 3 text

{ } CC-BY-ND 4.0 Logstash quick intro Collect, parse and store logs 3

Slide 4

Slide 4 text

{ } CC-BY-ND 4.0 Logstash quick intro • Logs? 4 ALL  THE  DATA  with  a  timestamp No  timestamp?  Hey!  
 we’ll  add  one  for  you.  

Slide 5

Slide 5 text

{ } CC-BY-ND 4.0 Logstash quick intro Transport & processing
 of
 streaming / continuous data 5

Slide 6

Slide 6 text

{ } CC-BY-ND 4.0 Logstash quick intro – Heterogenous formats and protocols – Unstructured format – Decentralized 6 Logging problems

Slide 7

Slide 7 text

{ } CC-BY-ND 4.0 Definitions • Event • Plugin (input, filter, output) • Pipeline 7

Slide 8

Slide 8 text

{ } CC-BY-ND 4.0 Logstash pipeline 8

Slide 9

Slide 9 text

{ } CC-BY-ND 4.0 Logstash pipeline 9 3 stages pipeline

Slide 10

Slide 10 text

{ } CC-BY-ND 4.0 Plugin architecture • ~200 plugins https://github.com/logstash-plugins • Input plugins: captures external data+format & transform it to logstash events • Filter plugins: process/transform events • Output plugins: send events to external destination & format 10

Slide 11

Slide 11 text

{ } CC-BY-ND 4.0 Plugin architecture • Codecs are plugins • Optional part of input and output plugins • encode and decode raw data that enter or exit the pipeline • Character encoding/transcoding into UTF-8 11 Codecs

Slide 12

Slide 12 text

{ } CC-BY-ND 4.0 Plugin architecture 12

Slide 13

Slide 13 text

{ } CC-BY-ND 4.0 Logstash pipeline 13 3 stages pipeline

Slide 14

Slide 14 text

{ } CC-BY-ND 4.0 Logstash pipeline 14 Internal queuing

Slide 15

Slide 15 text

{ } CC-BY-ND 4.0 Logstash pipeline 15 backpressure

Slide 16

Slide 16 text

{ } CC-BY-ND 4.0 Scaling out Logstash 16 shipper/indexer architecture

Slide 17

Slide 17 text

{ } CC-BY-ND 4.0 Scaling up the pipeline • Logstash pipeline is multithreaded • Each stage offers configurable concurrency options • Leverage multiple cores in host 17

Slide 18

Slide 18 text

{ } CC-BY-ND 4.0 Pipeline input stage 18

Slide 19

Slide 19 text

{ } CC-BY-ND 4.0 Pipeline input worker 19

Slide 20

Slide 20 text

{ } CC-BY-ND 4.0 Pipeline filter stage 20

Slide 21

Slide 21 text

{ } CC-BY-ND 4.0 Pipeline filter stage 21

Slide 22

Slide 22 text

{ } CC-BY-ND 4.0 Pipeline filter stage 22

Slide 23

Slide 23 text

{ } CC-BY-ND 4.0 Pipeline output stage 23

Slide 24

Slide 24 text

{ } CC-BY-ND 4.0 Pipeline output stage 24

Slide 25

Slide 25 text

{ } CC-BY-ND 4.0 Function compilation • Filter and output stages are compiled to a single function • Anyone knows or can guess why? 25

Slide 26

Slide 26 text

{ } CC-BY-ND 4.0 Function Compilation • Hint 26

Slide 27

Slide 27 text

{ } CC-BY-ND 4.0 Function Compilation • Conditionals • Message passing is not free 27

Slide 28

Slide 28 text

{ } CC-BY-ND 4.0 Logstash pipeline 28

Slide 29

Slide 29 text

{ } Thank you! Colin Surprenant, Elasticsearch Software Engineer colin.surprenant@elastic.co colinsurprenant

Slide 30

Slide 30 text

{ } This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0