Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Elasticsearch Ingest Processors
Alexander Reelsen
October 30, 2018
Technology
0
66
Elasticsearch Ingest Processors
A BoF session held at the elastic on tour event in Frankfurt in October 2018.
Alexander Reelsen
October 30, 2018
Tweet
Share
More Decks by Alexander Reelsen
See All by Alexander Reelsen
spinscale
0
71
spinscale
0
270
spinscale
1
52
spinscale
1
290
spinscale
4
640
spinscale
0
94
spinscale
1
320
spinscale
1
280
spinscale
1
9.6k
Other Decks in Technology
See All in Technology
hponka
0
1.5k
legalforce
PRO
0
170
halhira
1
110
tutsunom
1
160
kema1015
0
570
ishiitetsuji
0
110
comucal
PRO
0
300
junendo
0
160
jozono
6
1.2k
charity
10
13k
legalforce
PRO
0
160
ihcomega56
2
830
Featured
See All Featured
lemiorhan
628
47k
chriscoyier
779
240k
skipperchong
8
780
bkeepers
PRO
322
53k
trishagee
24
3k
holman
462
280k
reverentgeek
167
7.3k
bermonpainter
343
26k
paulrobertlloyd
73
1.5k
thoeni
3
660
bryan
99
11k
samanthasiow
58
6.4k
Transcript
Alexander Reelsen alex@elastic.co @spinscale Elasticsearch Ingest Processors Luca Wintergerst luca.wintergerst@elastic.co
@LucaWintergerst
‣ Update ‣ Writing your own processors ‣ Use-Cases ‣
Discussion Agenda
Update
‣ bytes (convert to human readable bytes) ‣ dissect (grok
without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors
‣ - drop processor to fully drop an event ‣
"drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors
‣ performance bump in geoip processor ‣ per processor metrics
‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others
‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI
Future
Writing your own
‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest
plugin
Use-Cases
… ask all the things! Discussion