Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Transcript

1. Alexander Reelsen alex@elastic.co @spinscale Elasticsearch Ingest Processors Luca Wintergerst luca.wintergerst@elastic.co @LucaWintergerst
2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣ Discussion Agenda
3. Update
4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors
5. ‣ - drop processor to fully drop an event ‣ "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors
6. ‣ performance bump in geoip processor ‣ per processor metrics ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others
7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI Future
8. Writing your own
9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest plugin
10. Use-Cases
11. … ask all the things! Discussion