Slide 1

Slide 1 text

Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected] @LucaWintergerst

Slide 2

Slide 2 text

‣ Update ‣ Writing your own processors ‣ Use-Cases ‣ Discussion Agenda

Slide 3

Slide 3 text

Update

Slide 4

Slide 4 text

‣ bytes (convert to human readable bytes) ‣ dissect (grok without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

Slide 5

Slide 5 text

‣ - drop processor to fully drop an event ‣ "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

Slide 6

Slide 6 text

‣ performance bump in geoip processor ‣ per processor metrics ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

Slide 7

Slide 7 text

‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI Future

Slide 8

Slide 8 text

Writing your own

Slide 9

Slide 9 text

‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest plugin

Slide 10

Slide 10 text

Use-Cases

Slide 11

Slide 11 text

… ask all the things! Discussion