Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch Ingest Processors

Avatar for Alexander Reelsen Alexander Reelsen
October 30, 2018
Technology
0
220

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.
Avatar for Alexander Reelsen

Alexander Reelsen

October 30, 2018
Tweet

More Decks by Alexander Reelsen

See All by Alexander Reelsen
Elasticsearch: From Keyword Search To Data Science
Avatar for Alexander Reelsen spinscale
0
140
Evolving Search at an ecommerce marketplace
Avatar for Alexander Reelsen spinscale
0
160
The new generation of data stores
Avatar for Alexander Reelsen spinscale
0
250
Search Evolution - Keeping up with the hype?
Avatar for Alexander Reelsen spinscale
0
380
Mirror mirror... what am I typing next?
Avatar for Alexander Reelsen spinscale
0
460
The New Generation of Data Stores
Avatar for Alexander Reelsen spinscale
0
230
Elasticsearch: Distributed Search Under the Hood
Avatar for Alexander Reelsen spinscale
0
160
Working distributed - but how?
Avatar for Alexander Reelsen spinscale
0
200
Implementing a custom aws lambda runtime using Crystal
Avatar for Alexander Reelsen spinscale
0
740

Other Decks in Technology

See All in Technology
技術職じゃない私がVibe Codingで感じた、AGIが身近になる未来
Avatar for ayane blueb
0
120
評価の納得感を2段階高める「構造化フィードバック」
Avatar for あろえ aloerina
1
140
「伝える」を加速させるCursor術
Avatar for Naomi Shiraishi naomix
0
610
Workflows から Agents へ ~ 生成 AI アプリの成長過程とアプローチ~
Avatar for Belong inc. belongadmin
2
140
堅牢な認証基盤の実現 TypeScriptで代数的データ型を活用する
Avatar for KAKEHASHI kakehashi
PRO
1
210
白金鉱業Meetup_Vol.19_PoCはデモで語れ！顧客の本音とインサイトを引き出すソリューション構築​
Avatar for BrainPad brainpadpr
2
200
Amplifyとゼロからはじめた AIコーディング 成果と展望
Avatar for k.masachika mkdev10
1
130
讓測試不再 BB！ 從 BDD 到 CI/CD， 不靠人力也能 MVP
Avatar for LINE Developers Taiwan line_developers_tw
PRO
0
110
上長や社内ステークホルダーに対する解像度を上げて、より良い補完関係を築く方法 / How-to-increase-resolution-and-build-better-complementary-relationships-with-your-bosses-and-internal-stakeholders
Avatar for MADOX madoxten
13
7.4k
自分を理解するAI時代の準備 〜マイプロフィールMCPの実装〜
Avatar for edo edo_m18
0
100
今からでも間に合う！ 生成AI「RAG」再入門 / Re-introduction to RAG in Generative AI
Avatar for Hideaki Aoyagi hideakiaoyagi
1
160
“プロダクトを好きになれるか“も QAエンジニア転職の大事な判断基準だと思ったの
Avatar for yuden tomodakengo
0
110

Featured

See All Featured
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
46
14k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
349
20k
Code Review Best Practice
Avatar for Trisha Gee trishagee
68
18k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
52
7.6k
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
329
21k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
6.6k
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
43
2.4k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
26
2.8k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
10
630
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
161
15k
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
179
9.8k

Transcript

  1. Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected]

    @LucaWintergerst

  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda

  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future

  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin

  10. Use-Cases

  11. … ask all the things! Discussion