Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Transcript

1. Alexander Reelsen
   alex@elastic.co
   @spinscale
   Elasticsearch
   Ingest Processors
   Luca Wintergerst
   luca.wintergerst@elastic.co
   @LucaWintergerst
   

   View Slide

2. ‣ Update
   ‣ Writing your own processors
   ‣ Use-Cases
   ‣ Discussion
   Agenda
   

   View Slide

3. Update
   

   View Slide

4. ‣ bytes (convert to human readable bytes)
   ‣ dissect (grok without regexes, much faster)
   ‣ pipeline processor, referring to other pipelines
   New processors
   

   View Slide

5. ‣ - drop processor to fully drop an event
   ‣ "drop" : { "if": "ctx.foo == 'bar'" }
   ‣ - scripting can invoke other processors
   ‣ "ctx.target_field = Processors.bytes(ctx.source_field)"
   ‣ if in every processor using scripting
   New processors
   

   View Slide

6. ‣ performance bump in geoip processor
   ‣ per processor metrics
   ‣ index default pipeline:
   ‣ settings.index.default_pipeline: "my_pipeline"
   Others
   

   View Slide

7. ‣ Aligning dissect filters in logstash/beats/ES
   ‣ https://github.com/elastic/dissect-specification
   ‣ UI
   Future
   

   View Slide

8. Writing your own
   

   View Slide

9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor
   ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect
   ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp
   Write your own ingest plugin
   

   View Slide

10. Use-Cases
    

    View Slide

11. … ask all the things!
    Discussion
    

    View Slide