Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Elasticsearch Ingest Processors

Avatar for Alexander Reelsen Alexander Reelsen
October 30, 2018
Technology
290
0

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Avatar for Alexander Reelsen

Alexander Reelsen

October 30, 2018

More Decks by Alexander Reelsen

See All by Alexander Reelsen
Understanding Apache Lucene - More than just full-text search
Avatar for Alexander Reelsen spinscale
0
210
Elasticsearch: From Keyword Search To Data Science
Avatar for Alexander Reelsen spinscale
0
230
Evolving Search at an ecommerce marketplace
Avatar for Alexander Reelsen spinscale
0
270
The new generation of data stores
Avatar for Alexander Reelsen spinscale
0
360
Search Evolution - Keeping up with the hype?
Avatar for Alexander Reelsen spinscale
0
450
Mirror mirror... what am I typing next?
Avatar for Alexander Reelsen spinscale
0
560
The New Generation of Data Stores
Avatar for Alexander Reelsen spinscale
0
320
Elasticsearch: Distributed Search Under the Hood
Avatar for Alexander Reelsen spinscale
0
220
Working distributed - but how?
Avatar for Alexander Reelsen spinscale
0
260

Other Decks in Technology

See All in Technology
2026_devsumi_ozono.pdf
Avatar for O3(ozono) o3
3
520
すぐできる衛星通信対応 あとは山奥に行くだけ
Avatar for tatetate55 tatetate55
0
130
山手線を徒歩で一周してわかった、 位置情報アプリは「足」が最強のデバッガー
Avatar for hinakko hinakko
0
160
【技術的負債conf】事業成長に伴う技術的負債の説明責任とAIによるモニタリング、認知的負債について
Avatar for Masato Ishigaki / 石垣雅人 i35_267
3
1.7k
データ界隈LT祭 第1回LT登壇
Avatar for タロウ taromatsui_cccmkhd
2
1.4k
10Xに技術的負債をもたらした「2つの境界の歪み」その構造と解消への営み
Avatar for 10xinc 10xinc
0
2.1k
負債のメタファと2026年 / Debt Metaphor in Agentic Engineering Age 202609 Edition
Avatar for Takuto WADA twada
PRO
9
4k
Claude Code本って、 読む必要あるの？
Avatar for Oikon oikon48
2
480
AIエージェントの権限管理 3:  Agentic RAG の Fine grained access control 編
Avatar for Renya Kujirada ren8k
0
110
Reactの設計論
Avatar for uhyo uhyo
24
14k
あけおめLINE 傾向とその対策
Avatar for nasa9084 nasa9084
0
130
2026-09-11 【Snowflake World Tour Tokyo 2026】Snowflakeを起点に、AI Agentが自律稼働し続ける未来へ / Driving AI Agents with Snowflake
Avatar for civitaspo civitaspo
0
700

Featured

See All Featured
Paper Plane (Part 1)
Avatar for Katie Cordina Lindsey katiecoart
PRO
2
11k
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
18k
4 Signs Your Business is Dying
Avatar for Josh Pigford shpigford
187
23k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
Avatar for Szymon Słowik szymonslowik
1
1.3k
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
Avatar for Helen Beal helenjbeal
1
390
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
9.2k
Designing Experiences People Love
Avatar for Jonathan Moore moore
143
24k
The AI Revolution Will Not Be Monopolized: How open-source beats economies of scale, even for LLMs
Avatar for Ines Montani inesmontani
PRO
3
3.7k
Efficient Content Optimization with Google Search Console & Apps Script
Avatar for Katarina Dahlin katarinadahlin
PRO
1
850
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
1
320
From π to Pie charts
Avatar for Rasagy Sharma rasagy
1
370

Transcript

  1. Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected]

    @LucaWintergerst

  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda

  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future

  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin

  10. Use-Cases

  11. … ask all the things! Discussion