Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearch Ingest Processors
Search
Alexander Reelsen
October 30, 2018
Technology
0
220
Elasticsearch Ingest Processors
A BoF session held at the elastic on tour event in Frankfurt in October 2018.
Alexander Reelsen
October 30, 2018
Tweet
Share
More Decks by Alexander Reelsen
See All by Alexander Reelsen
Elasticsearch: From Keyword Search To Data Science
spinscale
0
140
Evolving Search at an ecommerce marketplace
spinscale
0
160
The new generation of data stores
spinscale
0
250
Search Evolution - Keeping up with the hype?
spinscale
0
380
Mirror mirror... what am I typing next?
spinscale
0
460
The New Generation of Data Stores
spinscale
0
230
Elasticsearch: Distributed Search Under the Hood
spinscale
0
160
Working distributed - but how?
spinscale
0
200
Implementing a custom aws lambda runtime using Crystal
spinscale
0
740
Other Decks in Technology
See All in Technology
技術職じゃない私がVibe Codingで感じた、AGIが身近になる未来
blueb
0
120
評価の納得感を2段階高める「構造化フィードバック」
aloerina
1
140
「伝える」を加速させるCursor術
naomix
0
610
Workflows から Agents へ ~ 生成 AI アプリの成長過程とアプローチ~
belongadmin
2
140
堅牢な認証基盤の実現 TypeScriptで代数的データ型を活用する
kakehashi
PRO
1
210
白金鉱業Meetup_Vol.19_PoCはデモで語れ!顧客の本音とインサイトを引き出すソリューション構築
brainpadpr
2
200
Amplifyとゼロからはじめた AIコーディング 成果と展望
mkdev10
1
130
讓測試不再 BB! 從 BDD 到 CI/CD, 不靠人力也能 MVP
line_developers_tw
PRO
0
110
上長や社内ステークホルダーに対する解像度を上げて、より良い補完関係を築く方法 / How-to-increase-resolution-and-build-better-complementary-relationships-with-your-bosses-and-internal-stakeholders
madoxten
13
7.4k
自分を理解するAI時代の準備 〜マイプロフィールMCPの実装〜
edo_m18
0
100
今からでも間に合う! 生成AI「RAG」再入門 / Re-introduction to RAG in Generative AI
hideakiaoyagi
1
160
“プロダクトを好きになれるか“も QAエンジニア転職の大事な判断基準だと思ったの
tomodakengo
0
110
Featured
See All Featured
GraphQLとの向き合い方2022年版
quramy
46
14k
Reflections from 52 weeks, 52 projects
jeffersonlam
349
20k
Code Review Best Practice
trishagee
68
18k
We Have a Design System, Now What?
morganepeng
52
7.6k
Agile that works and the tools we love
rasmusluckow
329
21k
Automating Front-end Workflow
addyosmani
1370
200k
Music & Morning Musume
bryan
46
6.6k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
43
2.4k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.8k
Site-Speed That Sticks
csswizardry
10
630
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
For a Future-Friendly Web
brad_frost
179
9.8k
Transcript
Alexander Reelsen
[email protected]
@spinscale Elasticsearch Ingest Processors Luca Wintergerst
[email protected]
@LucaWintergerst
‣ Update ‣ Writing your own processors ‣ Use-Cases ‣
Discussion Agenda
Update
‣ bytes (convert to human readable bytes) ‣ dissect (grok
without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors
‣ - drop processor to fully drop an event ‣
"drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors
‣ performance bump in geoip processor ‣ per processor metrics
‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others
‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI
Future
Writing your own
‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest
plugin
Use-Cases
… ask all the things! Discussion