Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Elasticsearch Ingest Processors

Avatar for Alexander Reelsen Alexander Reelsen
October 30, 2018
Technology
300
0

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Avatar for Alexander Reelsen

Alexander Reelsen

October 30, 2018

More Decks by Alexander Reelsen

See All by Alexander Reelsen
Understanding Apache Lucene - More than just full-text search
Avatar for Alexander Reelsen spinscale
0
240
Elasticsearch: From Keyword Search To Data Science
Avatar for Alexander Reelsen spinscale
0
250
Evolving Search at an ecommerce marketplace
Avatar for Alexander Reelsen spinscale
0
310
The new generation of data stores
Avatar for Alexander Reelsen spinscale
0
380
Search Evolution - Keeping up with the hype?
Avatar for Alexander Reelsen spinscale
0
470
Mirror mirror... what am I typing next?
Avatar for Alexander Reelsen spinscale
0
570
The New Generation of Data Stores
Avatar for Alexander Reelsen spinscale
0
330
Elasticsearch: Distributed Search Under the Hood
Avatar for Alexander Reelsen spinscale
0
230
Working distributed - but how?
Avatar for Alexander Reelsen spinscale
0
270

Other Decks in Technology

See All in Technology
大江戸Ruby会議でデータベース クライアントの話をするのは 間違っているだろうか
Avatar for Yasuo Honda yahonda
1
110
Copilot Coworkで「みのるん式」やってみた
Avatar for みのるん minorun365
PRO
2
1.7k
田んぼの見回りをIoTで減らしたい!水位監視から電池駆動カメラまでの試行錯誤
Avatar for Katsuyoshi Ito itosoft
1
110
DBTS2026 盛り上がってますか！？Databricksが実現するAI時代のデータ基盤
Avatar for Akihiro Kuwano akuwano
1
200
TSKaigi Sendai 2026 - tsgo時代のTypeScript Playgroundはどう動くか
Avatar for teamLab teamlab
PRO
0
120
Bots in Kiro Crew (One point of contact for ongoing work) | Kiro: Zero to Hero 12
Avatar for Riku INADA inariku
PRO
1
120
プロダクションコードにおける encoding/json から encoding/json/v2 へのスムーズな移行
Avatar for SansanTech sansantech
PRO
1
250
Context Ontology Accelerator 入門 (業務データの意味を、AI エージェントに渡す)
Avatar for Riku INADA inariku
PRO
3
620
Kiro IDE 入門 (画面の各部から、頼み方、任せ方、戻し方まで) | 目指せ Kiro マスター 02
Avatar for Riku INADA inariku
PRO
4
200
Microsoft Entra Kerberos で実現するクラウド デバイス管理
Avatar for tamaiyutaro tamaiyutaro
2
660
Building Audio Plugin Ecosystem for Android in 2026
Avatar for Atsushi Eno atsushieno
2
160
Metadata - Section 3 : Dublin Core Standard · DC
Avatar for mohab soliman mohab1
0
120

Featured

See All Featured
Why You Should Never Use an ORM
Avatar for John Nunemaker jnunemaker
PRO
61
10k
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
Avatar for Akash Hashmi akashhashmi
0
550
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
31
7k
Code Review Best Practice
Avatar for Trisha Gee trishagee
74
20k
Are puppies a ranking factor?
Avatar for Jono Alderson jonoalderson
2
4.2k
First, design no harm
Avatar for Per Axbom axbom
PRO
2
1.4k
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
Avatar for Mfonobong Umondia mfonobong
2
630
Lightning Talk: Beautiful Slides for Beginners
Avatar for Ines Montani inesmontani
PRO
2
780
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
1
650
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
29
4.6k
From Legacy to Launchpad: Building Startup-Ready Communities
Avatar for Dug Song dugsong
0
380
SEOcharity - Dark patterns in SEO and UX: How to avoid them and build a more ethical web
Avatar for Sara Fernández Carmona sarafernandez
0
330

Transcript

  1. Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected]

    @LucaWintergerst

  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda

  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future

  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin

  10. Use-Cases

  11. … ask all the things! Discussion