Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Elasticsearch Ingest Processors
Alexander Reelsen
October 30, 2018
Technology
0
62
Elasticsearch Ingest Processors
A BoF session held at the elastic on tour event in Frankfurt in October 2018.
Alexander Reelsen
October 30, 2018
Tweet
Share
More Decks by Alexander Reelsen
See All by Alexander Reelsen
spinscale
0
64
spinscale
0
260
spinscale
1
50
spinscale
1
280
spinscale
4
620
spinscale
0
93
spinscale
1
310
spinscale
1
280
spinscale
1
9.5k
Other Decks in Technology
See All in Technology
yosshi_
2
310
shoken
0
110
hololab
0
340
clustervr
0
170
noir_neo
0
130
yosuke_matsuura
PRO
0
3.6k
shirayanagiryuji
0
2.2k
clustervr
0
200
hhiroshell
9
490
nihonbuson
2
2k
kawaguti
0
120
chaspy
1
410
Featured
See All Featured
iamctodd
17
1.9k
dotmariusz
94
5.1k
jonrohan
1021
380k
roundedbygravity
241
21k
brad_frost
156
6.4k
tenderlove
52
3.4k
maltzj
500
36k
thoeni
4
550
rocio
155
11k
samlambert
237
9.9k
eitanlees
111
9.9k
jrom
114
7.1k
Transcript
Alexander Reelsen alex@elastic.co @spinscale Elasticsearch Ingest Processors Luca Wintergerst luca.wintergerst@elastic.co
@LucaWintergerst
‣ Update ‣ Writing your own processors ‣ Use-Cases ‣
Discussion Agenda
Update
‣ bytes (convert to human readable bytes) ‣ dissect (grok
without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors
‣ - drop processor to fully drop an event ‣
"drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors
‣ performance bump in geoip processor ‣ per processor metrics
‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others
‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI
Future
Writing your own
‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest
plugin
Use-Cases
… ask all the things! Discussion