$30 off During Our Annual Pro Sale. View Details »

Elasticsearch Ingest Processors

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Alexander Reelsen

October 30, 2018
Tweet

More Decks by Alexander Reelsen

Other Decks in Technology

Transcript

  1. Alexander Reelsen alex@elastic.co @spinscale Elasticsearch Ingest Processors Luca Wintergerst luca.wintergerst@elastic.co

    @LucaWintergerst
  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda
  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors
  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors
  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others
  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future
  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin
  10. Use-Cases

  11. … ask all the things! Discussion