Elasticsearch Ingest Processors

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

D5cd900453405c985e97c63e9f92061d?s=128

Alexander Reelsen

October 30, 2018
Tweet

Transcript

  1. Alexander Reelsen alex@elastic.co @spinscale Elasticsearch Ingest Processors Luca Wintergerst luca.wintergerst@elastic.co

    @LucaWintergerst
  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda
  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors
  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors
  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others
  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future
  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin
  10. Use-Cases

  11. … ask all the things! Discussion