Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch Ingest Processors

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Alexander Reelsen

October 30, 2018
Tweet

More Decks by Alexander Reelsen

Other Decks in Technology

Transcript

  1. Alexander Reelsen
    [email protected]
    @spinscale
    Elasticsearch
    Ingest Processors
    Luca Wintergerst
    [email protected]
    @LucaWintergerst

    View Slide

  2. ‣ Update
    ‣ Writing your own processors
    ‣ Use-Cases
    ‣ Discussion
    Agenda

    View Slide

  3. Update

    View Slide

  4. ‣ bytes (convert to human readable bytes)
    ‣ dissect (grok without regexes, much faster)
    ‣ pipeline processor, referring to other pipelines
    New processors

    View Slide

  5. ‣ - drop processor to fully drop an event
    ‣ "drop" : { "if": "ctx.foo == 'bar'" }
    ‣ - scripting can invoke other processors
    ‣ "ctx.target_field = Processors.bytes(ctx.source_field)"
    ‣ if in every processor using scripting
    New processors

    View Slide

  6. ‣ performance bump in geoip processor
    ‣ per processor metrics
    ‣ index default pipeline:
    ‣ settings.index.default_pipeline: "my_pipeline"
    Others

    View Slide

  7. ‣ Aligning dissect filters in logstash/beats/ES
    ‣ https://github.com/elastic/dissect-specification
    ‣ UI
    Future

    View Slide

  8. Writing your own

    View Slide

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor
    ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect
    ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp
    Write your own ingest plugin

    View Slide

  10. Use-Cases

    View Slide

  11. … ask all the things!
    Discussion

    View Slide