Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch Ingest Processors

Avatar for Alexander Reelsen Alexander Reelsen
October 30, 2018
Technology
280
0

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Avatar for Alexander Reelsen

Alexander Reelsen

October 30, 2018

More Decks by Alexander Reelsen

See All by Alexander Reelsen
Understanding Apache Lucene - More than just full-text search
Avatar for Alexander Reelsen spinscale
0
200
Elasticsearch: From Keyword Search To Data Science
Avatar for Alexander Reelsen spinscale
0
220
Evolving Search at an ecommerce marketplace
Avatar for Alexander Reelsen spinscale
0
270
The new generation of data stores
Avatar for Alexander Reelsen spinscale
0
330
Search Evolution - Keeping up with the hype?
Avatar for Alexander Reelsen spinscale
0
440
Mirror mirror... what am I typing next?
Avatar for Alexander Reelsen spinscale
0
540
The New Generation of Data Stores
Avatar for Alexander Reelsen spinscale
0
300
Elasticsearch: Distributed Search Under the Hood
Avatar for Alexander Reelsen spinscale
0
200
Working distributed - but how?
Avatar for Alexander Reelsen spinscale
0
260

Other Decks in Technology

See All in Technology
世界、断片、モデル。そして理解
Avatar for SAKO Hiroshi ardbeg1958
1
130
AI、CDK と協働する Full TypeScript アプリケーション開発 / Full TypeScript Application with AI and CDK
Avatar for ギークプラス ソフトウェア事業部 geekplus_tech
2
410
個人開発で育てる「大規模設計の苗床」 - AI時代の1人開発から始める業務への知識接続 / The Seedbed for Large-Scale Design - From AI-Era Solo Projects to Professional Knowledge
Avatar for 株式会社ビットキー / Bitkey Inc. bitkey
PRO
1
280
AI時代の闇と光
Avatar for tatsuya1970 tatsuya1970
0
110
Data + AI Summit 2026 イベントレポート： 「AIがビジネスで意思決定するデータ基盤」へ
Avatar for KJ(Kenji Matsuda) nek0128
0
290
ファミコンでPHPを動かす / PHP on the Famicom
Avatar for HASEGAWA Tomoki tomzoh
2
480
副作用のある Lambda でも Lambda Power Tuning は使えるのか / lambda-power-tuning-side-effects
Avatar for koukihosaka koukihosaka
1
120
凡エンジニアがこの先生きのこるためには。〜TypeScript完全に理解したい〜
Avatar for Ryotaro Harada alchemy1115
2
320
「最後に責任を取るのはチーム」— 人間のPRレビューを最小化してアップデートしたメンタルモデル
Avatar for Jotaro Nishime jnishime_dresscode
0
940
タスクの複雑さでモデルを選ぶ ── Thompson Samplingで動かす“トークン/コスト最適化
Avatar for satoh-y-0323 satohy0323
0
570
「AIに依存している」と 「AIを使いこなしている」の違い
Avatar for Kensuke Yasuma k8yasuma
0
120
AI x 開発生産性を取り巻く予算戦略と投資対効果
Avatar for Masato Ishigaki / 石垣雅人 i35_267
1
280

Featured

See All Featured
Discover your Explorer Soul
Avatar for Emna emna__ayadi
2
1.2k
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
57
7k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
37
7.4k
SERP Conf. Vienna - Web Accessibility: Optimizing for Inclusivity and SEO
Avatar for Sara Fernández Carmona sarafernandez
2
1.5k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
85
9.6k
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
230k
Building a A Zero-Code AI SEO Workflow
Avatar for Ian Lurie portentint
PRO
0
630
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
59
6.8k
Money Talks: Using Revenue to Get Sh*t Done
Avatar for Nikki Halliwell nikkihalliwell
0
390
Organizational Design Perspectives: An Ontology of Organizational Design Elements
Avatar for Dr. Kim W Petersen kimpetersen
PRO
1
770
Joys of Absence: A Defence of Solitary Play
Avatar for Sebastian Deterding codingconduct
1
410

Transcript

  1. Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected]

    @LucaWintergerst

  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda

  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future

  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin

  10. Use-Cases

  11. … ask all the things! Discussion