Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch Ingest Processors

Avatar for Alexander Reelsen Alexander Reelsen
October 30, 2018
Technology
0
250

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Avatar for Alexander Reelsen

Alexander Reelsen

October 30, 2018
Tweet

More Decks by Alexander Reelsen

See All by Alexander Reelsen
Elasticsearch: From Keyword Search To Data Science
Avatar for Alexander Reelsen spinscale
0
190
Evolving Search at an ecommerce marketplace
Avatar for Alexander Reelsen spinscale
0
230
The new generation of data stores
Avatar for Alexander Reelsen spinscale
0
280
Search Evolution - Keeping up with the hype?
Avatar for Alexander Reelsen spinscale
0
400
Mirror mirror... what am I typing next?
Avatar for Alexander Reelsen spinscale
0
500
The New Generation of Data Stores
Avatar for Alexander Reelsen spinscale
0
260
Elasticsearch: Distributed Search Under the Hood
Avatar for Alexander Reelsen spinscale
0
170
Working distributed - but how?
Avatar for Alexander Reelsen spinscale
0
230
Implementing a custom aws lambda runtime using Crystal
Avatar for Alexander Reelsen spinscale
0
850

Other Decks in Technology

See All in Technology
業務の煩悩を祓うAI活用術108選 / AI 108 Usages
Avatar for 株式会社スマートバンク smartbank
9
18k
2025-12-27 Claude CodeでPRレビュー対応を効率化する@機械学習社会実装勉強会第54回
Avatar for Naka Masato nakamasato
4
1.3k
AWS Lambda durable functions を使って AWS Lambda の15分の壁を超えてみよう
Avatar for MatsuzawaTakeshi matsuzawatakeshi
0
110
Autonomous Database - Dedicated 技術詳細 / adb-d_technical_detail_jp
Avatar for oracle4engineer oracle4engineer
PRO
5
11k
20251219 OpenIDファウンデーション・ジャパン紹介 / OpenID Foundation Japan Intro
Avatar for OpenID Foundation Japan oidfj
0
600
まだ間に合う！ Agentic AI on AWSの現在地をやさしく一挙おさらい
Avatar for みのるん minorun365
19
3.3k
Microsoft Agent Frameworkの可観測性
Avatar for tomokusaba tomokusaba
1
120
BidiAgent と Nova 2 Sonic から​考える音声 AI について
Avatar for Yuuki Yamashita yama3133
2
130
Snowflake導入から1年、LayerXのデータ活用の現在 / One Year into Snowflake: How LayerX Uses Data Today
Avatar for Civitaspo civitaspo
0
2.6k
MariaDB Connector/C のcaching_sha2_passwordプラグインの仕様について
Avatar for kubo ayumu boro1234
0
1.1k
日本の AI 開発と世界の潮流 / GenAI Development in Japan
Avatar for Yoshitaka Haribara hariby
2
710
MySQLのSpatial(GIS)機能をもっと充実させたい ～ MyNA望年会2025LT
Avatar for sakaik sakaik
0
180

Featured

See All Featured
The agentic SEO stack - context over prompts
Avatar for schlessera schlessera
0
570
The Director’s Chair: Orchestrating AI for Truly Effective Learning
Avatar for Mike Taylor tmiket
1
69
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
PRO
0
42
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
7k
16th Malabo Montpellier Forum Presentation
Avatar for AKADEMIYA2063 akademiya2063
PRO
0
32
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
57
6.7k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
31
6.5k
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
67
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
Avatar for Mine Cetinkaya-Rundel minecr
0
97
The World Runs on Bad Software
Avatar for Brandon Keepers bkeepers
PRO
72
12k
Beyond borders and beyond the search box: How to win the global "messy middle" with AI-driven SEO
Avatar for David Carrasco davidcarrasco
0
26

Transcript

  1. Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected]

    @LucaWintergerst

  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda

  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future

  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin

  10. Use-Cases

  11. … ask all the things! Discussion