Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Elasticsearch Ingest Processors

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Alexander Reelsen Alexander Reelsen
October 30, 2018
Technology
300
0

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Avatar for Alexander Reelsen

Alexander Reelsen

October 30, 2018

More Decks by Alexander Reelsen

See All by Alexander Reelsen
Understanding Apache Lucene - More than just full-text search
Avatar for Alexander Reelsen spinscale
0
220
Elasticsearch: From Keyword Search To Data Science
Avatar for Alexander Reelsen spinscale
0
240
Evolving Search at an ecommerce marketplace
Avatar for Alexander Reelsen spinscale
0
280
The new generation of data stores
Avatar for Alexander Reelsen spinscale
0
360
Search Evolution - Keeping up with the hype?
Avatar for Alexander Reelsen spinscale
0
450
Mirror mirror... what am I typing next?
Avatar for Alexander Reelsen spinscale
0
560
The New Generation of Data Stores
Avatar for Alexander Reelsen spinscale
0
320
Elasticsearch: Distributed Search Under the Hood
Avatar for Alexander Reelsen spinscale
0
220
Working distributed - but how?
Avatar for Alexander Reelsen spinscale
0
270

Other Decks in Technology

See All in Technology
今さら理解するウェブフォントver.2026
Avatar for takanorip takanorip
3
540
ハードウェアコンペでもAI駆動開発が進んでいる話
Avatar for misoji engineer iotengineer22
1
660
12 個 Tool と、 4,096 Token
Avatar for ラワンアウンピョウ hlum
0
110
More Freedom on the Same Shared GPU Cluster: A Small Team’s Experience with vCluster
Avatar for NTT docomo Business nttcom
0
140
継続的なAIコスト最適化
Avatar for Atsushi Tanaka bgpat
1
270
Deep Data Security 機能解説
Avatar for oracle4engineer oracle4engineer
PRO
2
750
しろおびから始める！ログ活用の第一歩
Avatar for Shumei Ito shumei_ito
0
820
スキルを作る、その前に！複数人で使われるスキルを 作るためのプロセス
Avatar for Junki Furukawa junkifurukawa
2
370
おそらく日本で唯一のDevRelインターン生として
Avatar for husensan husengs7
0
170
自作バイナリを動かす「入口」8選
Avatar for Takumi Sueda puhitaku
1
1.2k
購入ドメインでの課題と取り組み
Avatar for ykagano ykagano
0
150
The seven pitfalls of AI (revised version)
Avatar for Uwe Friedrichsen ufried
0
310

Featured

See All Featured
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
9
1.2k
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
247
13k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
76
5.4k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
85
9.7k
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
Avatar for Aleyda Solis aleyda
2
12k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
672
58k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.5k
AI in Enterprises - Java and Open Source to the Rescue
Avatar for ivargrimstad ivargrimstad
0
1.5k
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
4
1.2k
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
2
530
Paper Plane (Part 1)
Avatar for Katie Cordina Lindsey katiecoart
PRO
2
11k
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
280
24k

Transcript

  1. Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected]

    @LucaWintergerst

  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda

  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future

  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin

  10. Use-Cases

  11. … ask all the things! Discussion