Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch Ingest Processors

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Alexander Reelsen Alexander Reelsen
October 30, 2018
Technology
0
270

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Avatar for Alexander Reelsen

Alexander Reelsen

October 30, 2018
Tweet

More Decks by Alexander Reelsen

See All by Alexander Reelsen
Understanding Apache Lucene - More than just full-text search
Avatar for Alexander Reelsen spinscale
0
140
Elasticsearch: From Keyword Search To Data Science
Avatar for Alexander Reelsen spinscale
0
200
Evolving Search at an ecommerce marketplace
Avatar for Alexander Reelsen spinscale
0
250
The new generation of data stores
Avatar for Alexander Reelsen spinscale
0
300
Search Evolution - Keeping up with the hype?
Avatar for Alexander Reelsen spinscale
0
420
Mirror mirror... what am I typing next?
Avatar for Alexander Reelsen spinscale
0
520
The New Generation of Data Stores
Avatar for Alexander Reelsen spinscale
0
280
Elasticsearch: Distributed Search Under the Hood
Avatar for Alexander Reelsen spinscale
0
180
Working distributed - but how?
Avatar for Alexander Reelsen spinscale
0
250

Other Decks in Technology

See All in Technology
AI時代のシステム開発者の仕事_20260328
Avatar for 閃利㈱河村 純 sengtor
0
310
Cursor Subagentsはいいぞ
Avatar for Yuji Yamaguchi yug1224
2
120
TUNA Camp 2026 京都Stage ヒューリスティックアルゴリズム入門
Avatar for terry-u16 terryu16
0
630
互換性のある（らしい）DBへの移行など考えるにあたってたいへんざっくり
Avatar for Takanori Sejima sejima
PRO
0
350
SaaSに宿る21g
Avatar for Yamakan kanyamaguc
2
180
「活動」は激変する。「ベース」は変わらない ～ 4つの軸で捉える_AI時代ソフトウェア開発マネジメント
Avatar for 辻裕士/sentokun sentokun
0
130
MIX AUDIO EN BROADCAST
Avatar for Erick Ralph Kionga ralpherick
0
130
AIエージェント勉強会第3回 エージェンティックAIの時代がやってきた
Avatar for Y.Miyado ymiya55
0
170
SSoT（Single Source of Truth）で「壊して再生」する設計
Avatar for かわうそ kawauso
2
400
Zephyr(RTOS)でOpenPLCを実装してみた
Avatar for misoji engineer iotengineer22
0
160
AIエージェント時代に必要な オペレーションマネージャーのロールとは
Avatar for KentaroFujii kentarofujii
0
230
Sansanの認証基盤を支えるアーキテクチャとその振り返り
Avatar for SansanTech sansantech
PRO
1
120

Featured

See All Featured
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
249
1.3M
SEO in 2025: How to Prepare for the Future of Search
Avatar for Michael King ipullrank
3
3.4k
The Curse of the Amulet
Avatar for Matthew Lei leimatthew05
1
11k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
231
22k
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
281
24k
Abbi's Birthday
Avatar for Violet Bock coloredviolet
2
6k
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
How to Align SEO within the Product Triangle To Get 
Buy-In & Support - #RIMC
Avatar for Aleyda Solis aleyda
1
1.5k
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
160
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
Avatar for UX Y'all uxyall
0
910
The AI Revolution Will Not Be Monopolized: How open-source beats economies of scale, even for LLMs
Avatar for Ines Montani inesmontani
PRO
3
3.2k
Leveraging Curiosity to Care for An Aging Population
Avatar for Cassini Nazir cassininazir
1
200

Transcript

  1. Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected]

    @LucaWintergerst

  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda

  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future

  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin

  10. Use-Cases

  11. … ask all the things! Discussion